On my desktop app PoShowTube I did get this working but not from inside the JavaFX window, I ended up with 2 windows. One for the JavaFX controls and one for the normal processing window, not ideal but it works for me. If you look on the user guide on PoShowTube website you get the idea of what I ended up with.
In hindsight, if I started again from scratch I think I probably would have dived into OpenGL directly which is obviously far more difficult to master but probably better suited to one window with lots of controls.