Quantum computation using optical modes has been well-established in its ability to construct deep neural networks. These networks have been shown to be flexible both architecturally as well as in terms of the type of data being processed. We leverage this property of the Continuous-Variable (CV) model to construct stacked single mode networks that are shown to learn structured classical information, while placing no restrictions on the size of the network, and at the same time maintaining its complexity. The hallmark of the CV model is its ability to forge non-linear functions using a set of gates that allows it to remain completely unitary. The proposed model exemplifies that the appropriate photonic hardware can be integrated with present day optical communication systems to meet our information processing requirements. In this paper, using the Strawberry Fields software library on the MNIST dataset of hand-written digits, we demonstrate the adaptability of the network to learn classical information in a multitude of machine learning tasks to very large fidelities.