The ensemble Kalman filter (EnKF) is a widely used metheodology for data assimilation problems and has been recently generalized to inverse problems, known as ensemble Kalman inversion (EKI). We view the method as a derivative free optimization method for a least-squares misfit functional and we present various variants of the scheme such as regularized EKI methods. This opens up the perspective to use the method in various areas of applications such as imaging, groundwater flow problems, biological problems as well as in the context of the training of neural networks. In particular, we will present application of the EKI to recent machine learning approaches, where we consider the incorporation of neural networks into inverse problems. We replace the complex forward model by a neural network acting as a physics-informed surrogate model, which will be trained in a one-shot fashion. This means we train the unknown parameter and the neural network at once, i.e. the neural network is only trained for the underlying unknown parameter. We connect the neural network based one-shot formulation to the Bayesian approach for inverse problems and apply the ensemble Kalman inversion in order to solve the optimization problem. Furthermore, we provide numerical experiments to highlight the promising direction of neural network based one-shot formulation together with the application of the ensemble Kalman inversion.