tensorflow中next_batch的具体使用

本文介绍了tensorflow中next_batch的具体使用，分享给大家，具体如下：

此处给出了几种不同的next_batch方法，该文章只是做出代码片段的解释，以备以后查看：

def next_batch(self, batch_size, fake_data=False):
 """Return the next `batch_size` examples from this data set."""
 if fake_data:
  fake_image = [1] * 784
  if self.one_hot:
   fake_label = [1] + [0] * 9
  else:
   fake_label = 0
  return [fake_image for _ in xrange(batch_size)], [
    fake_label for _ in xrange(batch_size)
  ]
 start = self._index_in_epoch
 self._index_in_epoch += batch_size
 if self._index_in_epoch > self._num_examples: # epoch中的句子下标是否大于所有语料的个数，如果为True,开始新一轮的遍历
  # Finished epoch
  self._epochs_completed += 1
  # Shuffle the data
  perm = numpy.arange(self._num_examples) # arange函数用于创建等差数组
  numpy.random.shuffle(perm) # 打乱
  self._images = self._images[perm]
  self._labels = self._labels[perm]
  # Start next epoch
  start = 0
  self._index_in_epoch = batch_size
  assert batch_size <= self._num_examples
 end = self._index_in_epoch
 return self._images[start:end], self._labels[start:end]

该段代码摘自mnist.py文件，从代码第12行start = self._index_in_epoch开始解释，_index_in_epoch-1是上一次batch个图片中最后一张图片的下边，这次epoch第一张图片的下标是从 _index_in_epoch开始，最后一张图片的下标是_index_in_epoch+batch, 如果 _index_in_epoch 大于语料中图片的个数，表示这个epoch是不合适的，就算是完成了语料的一遍的遍历，所以应该对图片洗牌然后开始新一轮的语料组成batch开始

def ptb_iterator(raw_data, batch_size, num_steps):
"""Iterate on the raw PTB data.

This generates batch_size pointers into the raw PTB data, and allows
minibatch iteration along these pointers.

Args:
 raw_data: one of the raw data outputs from ptb_raw_data.
 batch_size: int, the batch size.
 num_steps: int, the number of unrolls.

Yields:
 Pairs of the batched data, each a matrix of shape [batch_size, num_steps].
 The second element of the tuple is the same data time-shifted to the
 right by one.

Raises:
 ValueError: if batch_size or num_steps are too high.
"""
raw_data = np.array(raw_data, dtype=np.int32)

data_len = len(raw_data)
batch_len = data_len // batch_size #有多少个batch
data = np.zeros([batch_size, batch_len], dtype=np.int32) # batch_len 有多少个单词
for i in range(batch_size): # batch_size 有多少个batch
 data[i] = raw_data[batch_len * i:batch_len * (i + 1)]

epoch_size = (batch_len - 1) // num_steps # batch_len 是指一个batch中有多少个句子
#epoch_size = ((len(data) // model.batch_size) - 1) // model.num_steps # // 表示整数除法
if epoch_size == 0:
 raise ValueError("epoch_size == 0, decrease batch_size or num_steps")

for i in range(epoch_size):
 x = data[:, i*num_steps:(i+1)*num_steps]
 y = data[:, i*num_steps+1:(i+1)*num_steps+1]
 yield (x, y)

第三种方式：

 def next(self, batch_size):
   """ Return a batch of data. When dataset end is reached, start over.
   """
   if self.batch_id == len(self.data):
     self.batch_id = 0
   batch_data = (self.data[self.batch_id:min(self.batch_id +
                        batch_size, len(self.data))])
   batch_labels = (self.labels[self.batch_id:min(self.batch_id +
                        batch_size, len(self.data))])
   batch_seqlen = (self.seqlen[self.batch_id:min(self.batch_id +
                        batch_size, len(self.data))])
   self.batch_id = min(self.batch_id + batch_size, len(self.data))
   return batch_data, batch_labels, batch_seqlen

第四种方式：

def batch_iter(sourceData, batch_size, num_epochs, shuffle=True):
 data = np.array(sourceData) # 将sourceData转换为array存储
 data_size = len(sourceData)
 num_batches_per_epoch = int(len(sourceData) / batch_size) + 1
 for epoch in range(num_epochs):
   # Shuffle the data at each epoch
   if shuffle:
     shuffle_indices = np.random.permutation(np.arange(data_size))
     shuffled_data = sourceData[shuffle_indices]
   else:
     shuffled_data = sourceData

for batch_num in range(num_batches_per_epoch):
     start_index = batch_num * batch_size
     end_index = min((batch_num + 1) * batch_size, data_size)

yield shuffled_data[start_index:end_index]

迭代器的用法，具体学习Python迭代器的用法

另外需要注意的是，前三种方式只是所有语料遍历一次，而最后一种方法是，所有语料遍历了num_epochs次

来源：http://blog.csdn.net/appleml/article/details/57413615