python – Tensorflow Dataset.from_generator失败并出现pyfunc异常

我正在尝试使用tensorflow的每晚1.4,因为我需要Dataset.from_generator将一些可变长度的数据集放在一起.这个简单的代码(来自here的想法)：

import tensorflow as tf

Dataset = tf.contrib.data.Dataset
it2 = Dataset.range(5).make_one_shot_iterator()

def _dataset_generator():
    while True:
        try:
            try:
                get_next = it2.get_next()
                yield get_next
            except tf.errors.OutOfRangeError:
                continue
        except tf.errors.OutOfRangeError:
            return

# Dataset.from_generator need tensorflow > 1.3 !
das_dataset = Dataset.from_generator(_dataset_generator,
                                     output_types=(tf.float32, tf.float32))
das_dataset_it = das_dataset.make_one_shot_iterator()
with tf.Session() as sess:
    while True:
        print(sess.run(it2.get_next()))
        print(sess.run(das_dataset_it.get_next()))

失败了,相当神秘：

C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\Scripts\python.exe C:/Users/MrD/.PyCharm2017.2/config/scratches/scratch_55.py
0
2017-10-01 12:51:39.773135: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows\PY\35\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: 0-th value returned by pyfunc_0 is int32, but expects int64
     [[Node: PyFunc = PyFunc[Tin=[], Tout=[DT_INT64], token="pyfunc_0"]()]]
Traceback (most recent call last):
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
    return fn(*args)
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\_\Python35\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 467, in raise_exception_on_not_ok_status
    c_api.TF_GetCode(status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: 0-th value returned by pyfunc_0 is int32, but expects int64
     [[Node: PyFunc = PyFunc[Tin=[], Tout=[DT_INT64], token="pyfunc_0"]()]]
     [[Node: IteratorGetNext_1 = IteratorGetNext[output_shapes=[<unknown>, <unknown>], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](OneShotIterator_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/MrD/.PyCharm2017.2/config/scratches/scratch_55.py", line 24, in <module>
    print(sess.run(das_dataset_it.get_next()))
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Dropbox\_\PyCharmVirtual\TF-NIGHTLY\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 0-th value returned by pyfunc_0 is int32, but expects int64
     [[Node: PyFunc = PyFunc[Tin=[], Tout=[DT_INT64], token="pyfunc_0"]()]]
     [[Node: IteratorGetNext_1 = IteratorGetNext[output_shapes=[<unknown>, <unknown>], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](OneShotIterator_1)]]

Process finished with exit code 1

注意发电机工作正常：

with tf.Session() as sess:
    for k in _dataset_generator():
        print(sess.run(k))

打印：

0
1
2
3
4
Traceback (most recent call last):
...
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
     [[Node: IteratorGetNext_5 = IteratorGetNext[output_shapes=[[]], output_types=[DT_INT64], _device="/job:localhost/replica:0/task:0/cpu:0"](OneShotIterator)]]

正如所料.

这是一个错误,一个缺失的功能还是我误解了一些东西？

解决方法:

Dataset.from_generator()方法旨在将非TensorFlow Python代码连接到tf.data输入管道.例如,您可以从生成器生成简单的Python对象(例如int和str对象),列表或NumPy数组,并将它们转换为TensorFlow值.

但是,在您的示例代码中,您将得到it.get_next()的结果,它是一个tf.Tensor对象.这不受支持.
 如果需要在不同的数据集中捕获迭代器,可以在虚拟数据集上使用Dataset.map(),如下所示：

import tensorflow as tf

Dataset = tf.contrib.data.Dataset
it2 = Dataset.range(5).make_one_shot_iterator()

das_dataset = Dataset.from_tensors(0).repeat().map(lambda _: it2.get_next())
das_dataset_it = das_dataset.make_one_shot_iterator()
with tf.Session() as sess:
    while True:
        print(sess.run(it2.get_next()))
        print(sess.run(das_dataset_it.get_next()))

标签：python,tensorflow,generator,yield,tensorflow-datasets
来源： https://codeday.me/bug/20190823/1698895.html