When fd_close() is closing the socket, it will also:
After a short write, the portion that was not written is saved in the output buffer of the fd and the fd is watched for P_POLLOUT. When it becomes writable again, the buffer is written out first. If the buffer is empty, and the fd is not paused, unblocking is backpromoted.
Read buffer is used only in a corner case: a source fd of an RP got closed, constantly sending HUPs (through poll). In this case the process has already closed the fd, and data is buffered in the kernel, probably available immediately. Thus when HUP is received on a source FD, plumb will read all data up to eof into the the read buffer. This will not increase the amount of buffering done to the data at the end, merely changes the location of the buffer from the kernel to plumb.
As soon as the source FD gets unblocked, the buffered data is flushed first.
An interesting corner case is lurking in shutdown: when the last sticky process quits, there may be some HUP'd source FDs (especially from the given process) staying around. Before plumb goes on to kill processes, it first loops until all such read buffers are empty.
The second kind of buffer, the write buffer is used to handle short writes: remaining data after a short write has to be stored and written first, when the sink FD is available for POLLOUT again.
A corner case is planted here as well: when a VP tries to write to an already blocked sink FD, those write attempts are also appended to the write buffer. This should not happen normally, as upon the first short write a block is back-promoted to stop all sources attempting to feed more data into the network. However, an "RP1 | VP1 | VP2 | RP2" chain with a VP1 splitting the stream into short records, VP2 inserting considerable extra data (prefixing, suffixing, etc) will trigger this case. When the first short write happens at RP2, a block is back-promoted, through VP2->VP1 to RP1, which will stop feeding data in, but VP1 may already have multiple new records prepared that needs to be sent through before VP1 returns. This data will be transformed by RP2 and will end up in the write buffer of RP2.
This does not mean plumb buffers more than it should: all the extra data appended to the write buffer that way is still from the single last read() from RP1 - the extra appending happens only because the way VPs split and grow data.
Another corner case, which truely appends more data, sort of ignoring a block, is a "VP | RP" chain from a VP that emits data without reading it from source FDs (e.g. timer, event). In other words, thw actual blocking never happens at a VP but at an RP, which means if the final source of the data is a VP, it can not be blocked. In extreme situations this may cause plumb to grow very large write buffers collecting huge amount of VP-sourced data!
When a flush is executed, parsing stops and the remaining stages are run on the script seen so far, then the binary script buffer is deleted and parsing continues from after the flush the same way as in the beginning of a new file.
Only after finishing with start, data flow may start (P_poll called). This allows ->start of VPs to change piping inside the transaction without having to worry about buffers and partial lines: there's no data read from any of the RPs yet. Of course this is not true for processes created by earlier transactions.
It is also guaranteed that by the time ->start is called, all objects of the transaction are already created.
Virtual process [new] creates a new, independent transaction, which is not a subtransaction of any other transaction (has no parent). By the time ->init of [new] returns, the new transaction is already executed.
Each instance of [cmd] creates its own transaction with similar properties of the transactions of [new], except there is no implicit flush and the parsed but pending statements of the transaction is executed only when a flush command is parsed.