c-使用boost.process同时读取和写入孩子的stdio
作者：互联网
我正在尝试使用boost.process使用类似这样的东西来读写孩子的stdio：
boost::asio::io_service writeService, readService;
bp::async_pipe in{writeService};
bp::async_pipe out{readService};
bp::child process(CompressCmd.c_str(), bp::std_in < in, bp::std_out > out);
Buffer src;
src.reserve(4 * 1024 * 1024);
integer_type read = 0;
//std::atomic_int64_t totalWrite{0};
integer_type totalWrite = 0;
while (callback(CallbackActions::NeedMoreInput, src, read)) {
in.async_write_some(
boost::asio::buffer(src.data(), read),
[](const boost::system::error_code &e, std::size_t) { });
// written data is not important, that's why using same buffer
out.async_read_some(boost::asio::buffer(src.data(), src.capacity()),
[&](const boost::system::error_code &e,
std::size_t byte_transferred) { totalWrite += byte_transferred; });
}
writeService.run();
in.close();
readService.run();
所有读取和写入操作均被提名为成功,但是totalWrite的值完全不正确,例如报告为29356032,实际值应为50000000左右
我注意到程序在readService.run()冻结子级之后使用process.wait()终止程序,使用atomic int产生相同的行为,现在我实际上只需要知道实际写入了多少数据,这就是为什么我正在使用相同的缓冲区
解决方法:
>此模式：
while (callback(CallbackActions::NeedMoreInput, src, read)) {
in.async_write_some(...);
out.async_read_some(...);
}
是最有可能被误导的(异步操作总是立即返回,因此您只需继续添加更多的异步操作而不会给它们运行的机会).
>还有一个错误的事实,那就是您为管道提供了单独的服务,但是您将它们完全排除在外运行,因此在writeService完成之前将不会运行任何读取操作.
>原子类型被误导,因为无法从多个线程进行访问
>您想做什么？您保留了一个大缓冲区,但从未将任何数据放入其中(保留！=调整大小).因此,您只能希望什么都不写.
更具有讽刺意味的是,您正在完全相同的位置读取完全相同的缓冲区.但是,它立即为Undefined Behaviour¹,因为当您知道src.size()== 0时将其传递给src.capacity().
即使没有该错误,您如何也可以“同时”从内存中完全相同的字节进行读取和写入,仍然知道预期的结果是什么？
>您没有将自己的io_service传递给Boost Process
工作演示
这是一个工作示例.当然,我不得不猜测您实际上想做什么.
我选择使程序将自己的源(main.cpp)发送到stdin,并迭代读取stdout,记录total_received字节.然后打印退出代码和总数.
作为临时压缩器,我使用了’/usr/bin/xxd’,因为它可用并且甚至可以有用地打印到std :: cout进行调试.
Live on Coliru //麻烦Coliru
#include <boost/asio.hpp>
#include <boost/process.hpp>
#include <boost/process/async.hpp>
#include <iostream>
std::vector<char> read_file(std::string const&);
namespace bp = boost::process;
using boost::system::error_code;
using Loop = boost::function<void()>;
using Buffer = std::array<char, 4*1024>;
int main() {
boost::asio::io_service svc;
std::string const CompressCmd = "/usr/bin/xxd";
bp::async_pipe in{svc}, out{svc};
bp::child process(CompressCmd, bp::std_in < in, bp::std_out > out, svc);
auto data = read_file("main.cpp");
Loop read_loop, write_loop;
Buffer recv_buffer;
std::size_t total_received = 0;
read_loop = [&read_loop, &out, &recv_buffer, &total_received] {
out.async_read_some(boost::asio::buffer(recv_buffer),
[&](error_code ec, size_t transferred) {
std::cout << "ReadLoop: " << ec.message() << " got " << transferred << " bytes\n";
total_received += transferred;
if (!ec)
read_loop(); // continue reading
});
};
boost::asio::async_write(in, boost::asio::buffer(data),
[&](error_code ec, size_t transferred) {
std::cout << "WriteLoop: " << ec.message() << " done, " << transferred << " bytes\n";
in.close(ec);
std::cout << "WriteLoop: closed pipe (" << ec.message() << ")\n";
}); // async
read_loop(); // async
svc.run(); // Await all async operations
std::cout << "Process exitcode " << process.exit_code() << ", total_received=" << total_received << "\n";
}
#include <fstream>
#include <iterator>
std::vector<char> read_file(std::string const& fname) {
std::ifstream ifs(fname);
return {std::istreambuf_iterator<char>(ifs), {}};
}
印花
WriteLoop: Success done, 1787 bytes
WriteLoop: closed pipe (Success)
ReadLoop: Success got 4096 bytes
ReadLoop: Success got 3515 bytes
ReadLoop: End of file got 0 bytes
Process exitcode 0, total_received=7611
解释,简化
注意,我们所有的编写都没有循环.那是因为boost :: asio :: async_write是一个组合操作(它隐藏了循环).
同样,如果可以“负担得起”将接收到的全部数据存储在内存中,则可以通过使用boost :: asio :: streambuf并使用类似的组合操作来简化操作：
Live on Coliru //麻烦Coliru
#include <boost/asio.hpp>
#include <boost/process.hpp>
#include <boost/process/async.hpp>
#include <iostream>
std::vector<char> read_file(std::string const&);
namespace bp = boost::process;
using boost::system::error_code;
int main() {
boost::asio::io_service svc;
std::string const CompressCmd = "/usr/bin/xxd";
bp::async_pipe in{svc}, out{svc};
bp::child process(CompressCmd, bp::std_in < in, bp::std_out > out, svc);
auto data = read_file("main.cpp");
boost::asio::streambuf recv_buffer;
boost::asio::async_read(out, recv_buffer,
[&](error_code ec, size_t transferred) {
std::cout << "ReadLoop: " << ec.message() << " got " << transferred << " bytes\n";
});
boost::asio::async_write(in, boost::asio::buffer(data),
[&](error_code ec, size_t transferred) {
std::cout << "WriteLoop: " << ec.message() << " done, " << transferred << " bytes\n";
in.close(ec);
std::cout << "WriteLoop: closed pipe (" << ec.message() << ")\n";
}); // async
svc.run(); // Await all async operations
std::cout << "Process exitcode " << process.exit_code() << ", total_received=" << recv_buffer.size() << "\n";
}
#include <fstream>
#include <iterator>
std::vector<char> read_file(std::string const& fname) {
std::ifstream ifs(fname);
return {std::istreambuf_iterator<char>(ifs), {}};
}
Conversely, if you cannot afford to have all the data in memory before sending, you can create a loop to send input block-wise
两个具有延迟的异步循环
让我们做到这一点,并通过在写入每个块之前延迟一秒钟来使其更具娱乐性.您期望看到的是由于延迟而交替进行读/写：
Live On Coliru //可以在Coliru上运行
#include <boost/asio.hpp>
#include <boost/asio/high_resolution_timer.hpp>
#include <boost/process.hpp>
#include <boost/process/async.hpp>
#include <iostream>
#include <fstream>
namespace bp = boost::process;
using boost::system::error_code;
using namespace std::chrono_literals;
using Loop = boost::function<void()>;
using Buffer = std::array<char, 500>;
int main() {
boost::asio::io_service svc;
auto on_exit = [](int code, std::error_code ec) {
std::cout << "Exited " << code << " (" << ec.message() << ")\n";
};
std::string const CompressCmd = "/usr/bin/xxd";
bp::async_pipe in{svc}, out{svc};
bp::child process(CompressCmd, bp::std_in < in, bp::std_out > out, svc, bp::on_exit(on_exit));
Loop read_loop, write_loop;
Buffer recv_buffer;
std::size_t total_received = 0;
read_loop = [&read_loop, &out, &recv_buffer, &total_received] {
out.async_read_some(boost::asio::buffer(recv_buffer),
[&](error_code ec, size_t transferred) {
std::cout << "ReadLoop: " << ec.message() << " got " << transferred << " bytes\n";
total_received += transferred;
if (!ec)
read_loop(); // continue reading
});
};
std::ifstream ifs("main.cpp");
std::size_t total_written = 0;
Buffer send_buffer;
boost::asio::high_resolution_timer send_delay(svc);
write_loop = [&write_loop, &in, &ifs, &send_buffer, &total_written, &send_delay] {
if (!ifs.good())
{
error_code ec;
in.close(ec);
std::cout << "WriteLoop: closed stdin (" << ec.message() << ")\n";
return;
}
ifs.read(send_buffer.data(), send_buffer.size());
boost::asio::async_write(in, boost::asio::buffer(send_buffer.data(), ifs.gcount()),
[&](error_code ec, size_t transferred) {
std::cout << "WriteLoop: " << ec.message() << " sent " << transferred << " bytes\n";
total_written += transferred;
if (!ec) {
send_delay.expires_from_now(1s);
send_delay.async_wait([&write_loop](error_code ec) {
std::cout << "WriteLoop: send delay " << ec.message() << "\n";
if (!ec) write_loop(); // continue writing
});
}
});
};
read_loop(); // async
write_loop(); // async
svc.run(); // Await all async operations
std::cout << "Process exitcode " << process.exit_code() << ", total_received=" << total_received << "\n";
}
打印
WriteLoop: Success sent 500 bytes
WriteLoop: send delay Success
WriteLoop: Success sent 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 96 bytes
WriteLoop: send delay Success
WriteLoop: Success sent 500 bytes
WriteLoop: send delay Success
WriteLoop: Success sent 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 96 bytes
WriteLoop: send delay Success
WriteLoop: Success sent 500 bytes
WriteLoop: send delay Success
WriteLoop: Success sent 134 bytes
WriteLoop: send delay Success
WriteLoop: closed stdin (Success)
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 500 bytes
ReadLoop: Success got 22 bytes
Exited 0 (Success)
ReadLoop: End of file got 0 bytes
Process exitcode 0, total_received=11214
¹也许只是未指定,我现在不打算找出差异
标签：c,windows,boost,boost-asio,boost-process 来源： https://codeday.me/bug/20191013/1906769.html