If I understand the show I recently watched correctly, the new F-22 is equipped with a visual surveillance system that essentially allows the pilot to see “through” the aircraft by employing an array of cameras to observe clearly in all directions. Literally looking through the eye of the camera corresponding with their current ocular orientation, this technology is surely a milestone in the course of pilot battle awareness, correct?

Unless I’m incorrect, this technology at heart likely also takes merely a high-grade visual processing computer and a number of quality cameras to put into use. What serious cost barriers are in place to prevent the use of modern off-the-shelf technology or production facilities to mass-produce these devices fairly inexpensively? (American made, of course!)

How long until this technique or technology is employed for use in military vehicles, especially such as with tanks? Currently often looking through essentially periscopes, this ability to remain most accurately aware of a tank’s orientation with its surroundings and threats without being at times exposed external to the armor strikes me as just as similarly invaluable for armored vehicles as it is for jets.

Furthermore, vehicles such as the Stryker could be employed to drive through an area mounted with such equipment, the beauty of such a system being in that the observers do not actually have to be present nor in real-time. By quickly patrolling through areas, entire regions could be relatively quickly surveyed. Such vehicles could even be inconspicuously disguised rather than necessarily armored. This reconnaissance data could then be searched through by teams of virtual ‘passengers,’ soldiers riding through the reels and even stopping to take the time to more thoroughly investigate videos of suspicious events or developments. Beyond scouting and patrol, maps of cities and urban landscapes could be made that soldiers could explore safely prior to actual field deployment.

Anyway, these are my thoughts on the F-22 (or one of the numerous fascinating features it possesses) and also an alternative application that might be worth a hastened consideration and potential implementation. Minimizing soldier exposure and even being able to decrease the number of foreign deployed soldiers are both goals of much pressure these days. Being able to employ trained military observers from homeland bases rather than having to occupy them onsite in Iraq could be achieved for numerous missions we currently are engaged with. A postponed realistic engagement with Iraq the likely safest international scenario unfortunately, bringing large numbers of our troops home (or in the least never stop-losing another soldier again) could go far in alleviating much of the pressure the military is feeling to cease all operations within the Iraqi conflict zones.

Great analysis. Why do you ask? Just curious?

I'm a research assistant for the University of Iowa, working on similar technology. We're interested in ideas and possible ways to get this technology into the hands of more US military applications, as well as civilian apps. Your tank idea is great.

Also, if anyone has ideas for other uses for this, please let me know. If we could get grant money from the medical community, for instance, we could further develop the technology.

And from a supply-side economic standpoint, our economy could use more innovation that improves productivity and reduces costs of goods/services. But I digress.

Please keep the ideas coming, and let me know if you have ideas for how to move this technology forward, including contacts and likely sources of funding.

Honestly I kind of like the surveillance idea better then the tank implementation, as the tank employment seems simply inevitable. $30,000 for the observational upgrade ($4,000 for the cameras, $5,000 for the processing, $1,000 for an off-the-shelf terabytes hard drive, $5,000 for the HUD, $15,000 for sufficient armored mounting. Complete guess, but I’d love to hear more knowledgeable estimates.), $30,000 to substantially reduce the risk to soldiers’ lives and improve battlefield effectiveness, this is a clear cut win-win to me. And a heck of an investment if you can get in on production

The surveillance part though, this is the one that most interests me. A team going into a neighborhood that can first run a visual walkthrough of the area, exploring escape routes, likely threat zones, learning the ins and outs of the field, this seems even more invaluable in the long run. Being localized to an environment prior to ever having boots on the ground would minimize insurgency advantage gained through their own familiarization. Situational and environmental awareness could be greatly improved. A commander capable of simply stating a series of pre-trained rally points from a large selection rather than having to coax a coordinated movement, able to have faith in his units’ awareness of that theatre, might greatly increase the speed in which a combat zone can be dominated and contained.

Furthermore, intelligence analysis could be conducted in our own homeland rather than, for example, in Iraq. While requiring very substantial bandwidth, transmitting the audio-video recordings acquired in passes through regions would mean a database of behaviors, infrastructure, and concerns could be compiled that allows insights into civil behavior versus potential violent insurgency actions. Placing regions into context, heavy observation implementation could mean a set of observers would have a web of UAV aerial video, on the ground observances, hotspot surveillance or reconnaissance, and even more modern information based documentation, all woven into a GIS (Graphical information system) database that allows a fluid shifting and integration between different data resources. At its core a system such as this could improve context by allowing the observers to see the field from as many angles as possible as the events occur across a set frame of time. Piecing together as many streams of data into one geographical event that shows the behaviors emerging in delayed real time from each location, this seems to me to be the best way to track and isolate the organizing threats that are the true enemies we need most to cease to win a war against those cowardly waging terrorism warfare.

Admittedly I'm on a bit of a rant here, but I still see systems like this as likely to emerge soon as an effort to humanize arbitrary data is essential to fully implementing the resources that technology is now putting at our disposal. Placing things first and foremost into a more natural real time progression within a comprehensible and familiarized geographical terrain seems to me to be the most user friendly way to achieve this. This is readily within our capabilities today. I’m a bit far from the original ocular augmentation system but still, its just what I think is on the way
-Kenneth

Klyon,

Interesting analysis. Here's where we're at:

We need to get this technology into the military. We'll do that through the Navy (ONR) or wherever else we can. Because we know that, the more the information is used by one branch, the more it will catch on in others.

At OPL (the University of Iowa's Operator Performance Lab), we have a system to monitor operator performance. We've used this to evaluate operator cognitive state by plugging in different sensors, including EEG.

Anyway, we're still looking for departments in the military who are able to put our work to use. You can check out some of our stuff:
OPL website: http://www.ccad.uiowa.edu/opl/
CATS: http://www.ccad.uiowa.edu/opl/projects/statesensor/index.html
Link to Discovery Channel videos: http://www.ccad.uiowa.edu/opl/videos/

It certainly looks like an intriguing study, although I’m curious as to what more specifically you are trying to accomplish. The way I understand your system is the operator performance monitoring is used to isolate scenarios that most overwhelm pilots. Finding the specific features of these tasks that create the most distress, you then implement automated assistance either by micromanaging flight control or present the pilot the complex information as already categorized in a way to bypass the perplexing facets through compiling it in a more situation pertinent format, correct? Also, would I be correct in guessing minimally distracting methods of maintaining alertness during remedial periods is also a factor being looked into?

Alternatively, does the project make any headway into exploring more biofeedback oriented options? Conditioned responses through simulators could be employed to prime subconscious signifiers for mode selection, making flight micromanagement easier to integrate with the pilot given if the pilot is in fact the originator of the selected procedure. Being able to launch the vehicle into an advanced roll beyond a pilot’s traditional ability to handle G’s followed by a leveling out, such panicked but perhaps occasionally necessary maneuvers could be initiated almost instantaneously as initialized by the pilot’s cognizance of the desired action. I’d be curious as the difference in disorientation that occurs when a pilot is consciously aware of an emergency response prior to use versus the reorientation time necessary to regain awareness with the lack of forewarning, meaning a fraction of a second delay to check feedback might be ultimately more useful in a combat situation.

Stadler,
Has your organization considered applying your system on the more modern “Gamer” culture in evaluating their performance in similar tasks? Could the psychological responses occurring within individuals more comfortable with the mastery of a non-physically based stimulus structure compared with those obliged to mitigate inertial distractions be used to create a quantitative contrast between task performances associated with UMV controls versus in-object flight controlling? Btw, forgive my pedantic nature; I’m pretty notorious for it Anyway, it seems to me if you could demonstrate the improved or possibly detracted manipulation of off-location controllers as being potentially superior or not to that of onsite pilots, this may well be of future merit.

Furthermore, if a hybrid interface where a dampened physical simulator could be employed to demonstrate situational actuality for a distanced operator to optimize responsiveness possibly be developed, it seems your research might also have a longer reaching application if this is the case beyond the immediate development tasks I’ve seen. By collecting data relevant to the production of an optimized UMV user interface as well, the longevity of your project’s pertinence might similarly be improved. The slower flight and limited maneuverability of current aerial UMVs is bound to increase substantially, and finding systems more effective than the stationary user interfaces seen today is likewise bound to be needed. From the sounds of it, your Operator Performance Lab seems primed to deal with this sort of challenge.
Anyway, let me know if these thoughts are of any use to you.
-Kenneth

I'm glad your mind is going the same direction that we're going. What's your background?

To answer your question (kinda), our simulators are basically video games (I mean, that's the whole foundation for video games). The challenge is how to deliver feedback to the experienced pilot, piloting the UMV remotely.

It might be of interest to you, but we're working on a phase II SBIR for the Navy, investigating the effectiveness of different training systems (computer games, fixed-base simulators, moving-based simulators and real aircraft, like our L-29), as a function of fidelity. We're using physiological data from multiple sensors, along with mission performance data, to classify trainee performance and learning curves. One can speculate that the same toolset (flight hardware and sensor suite) can be used to better pilot UMVs offsite.

So, why do you know so much?

By an SBIR, does that mean your group is operating under a Small Business Innovation Research Program then similar to what NASA is awarding? If so, how much leeway do you guys have in redistributing your findings to the public sector versus its classification for strictly military use?

Anyway, I’m curious as to what form of information analysis your group is employing to isolate significant factors. Furthermore, are you involved in the actual programming side of the flight simulator training programs or simply evaluation and requests? The reason why I’m curious about this is ironically a family member of mine is a part of a group promoting the use of evolutionary computational systems, specifically through their program AutoCore. It’s designed to be easily linked into third party software. Furthermore, they have also worked in the past with the military developing an artillery training simulator as well as are currently working on complex visual identification tasks with them, among other projects.

Your project seems like something he may be quite interested in exploring ways they might be of use to you. It’s quite cutting edge techniques they’re focused around providing ways to apply, beyond the simulation space applications even more so they may be very practical in hastening the identification of significant sensor data such as connections betweens the various sources such as novel eye movement trends correlating with EEG results, etc. Here’s a link to a PDF about their military applications:
http://www.simudyne.com/resources/Military_and_Intelligence_Uses.pdf
Let me know if you and your group might be interested in hearing more or possibly speaking to someone and I’ll see what I can arrange.



Business aside, as for my own interest in these things, its mainly just in that I enjoy a good intellectual challenge, and the diversity of applications a problem that groups such as yours are facing certainly qualify as that sort of challenge. Background wise I’m pushing on being called a professional student, mainly as I have the habit of taking courses I’m interested in rather than what is needed to graduate. Combine that with not finding just what I’d like to do post-college and I’m seven years in and still a good two years plus from even my undergraduate. So, I’m fairly well versed in everything from philosophy to psychology to economics to political science, with a bit of engineering, programming, and even anthropology just to mix it up a bit. I’ve been accused of being more interested in trying to see the big picture rather than specializing into a set niche within which its practical to find work, which I can’t really say is far off base. It’s safe to say a think tank would be my dream job ;-]

In light of this current challenge though, I’ve been thinking more about possible methods and am curious as to if any of the following are being tested dealing with the employment of sound harmonics for data representation. Little bit of cut and paste here, hope you don’t mind:

1. Have sound harmonics been looked into as a way of conveying breaks from an equilibrium standard? Easily recognizable dissonant or chord combinations could be used to relate problems or even orientation. In example, using specific pitch combinations to relay the plane’s tilt, pan, etc to the pilot could be trained in a motion-based simulator. Using this training, a fixed-base operator might improve their awareness of orientation for remote flight control of vehicles without the cost of a bulky moving simulator.
2. Alternatively, could more complex information be conveniently conveyed in this manner? Recently a method for observing seismic data for risk zones has been developed using musical representation. Similarly, things such as engine or structural status might be presented to a pilot. A set cycling of patterns in bars which are harmonious representing optimal functioning could be used where specific forms of variation, trained beforehand for significance, could present status updates to the pilot without requiring them to take eyes off of their tracked target or cluttering the dashboard with additional gauges.
3. Furthermore, I recently watched a show where a savant pianist named Derek Paravicini was tested using EEG for musical recognition. By playing him songs he knew with or without alteration, they tried to ascertain his ability for recognizing these variations from expected. While his voiced responses were disconnected from his understanding due to his autism, the portion of the brain conducting the analysis showed clear signs for recognizing either conformity or variation. Now to the point: would it be possible to assign a musical representation to a tracked target’s flight pattern? By training the pilot to accurately understand the significance of the heard patterning, the pilot can ‘watch’ the target aircraft while being distracted by tasks such as designing their desired approach with the flight controls. Should the expected flight of the target shift from the expected course, could the interface or AI of the aircraft use this awareness of a divergence to improve its own use of its weapon systems or flight trajectory? By the pilot selecting the style of approach being attempted, the plane may be able to make immediate corrections where the pilot could take several seconds to adapt, especially when busied with micromanaging specifics. This probably isn’t the most modern of scenarios, but the idea of an AI within the interface maintaining an awareness of the situations alignment with pilot’s expectations using this sort of system is the main point here. There are probably other routes to this, but I’m curious about the utility of using an underemployed portion of the pilot’s attention instead of cluttering more traditional cognitive task processes with this sort of method.

Anyway, let me know if any of these thoughts are of potential use to your group or have in the least sparked some new brainstorming. Also, if you or your associates would be interested in hearing more about possible problem-solving that Simudyne might be able to bring to the table for you, I’d be more than happy to put you in contact with my cousin Justin Lyon. It’s been a little while since I’ve spoken to him so I can’t guarantee how free his schedule may be, but your project is right up their alley and I’m sure he’d be interested in at least taking the time to explore possible ways he might be of benefit to your efforts.
-Kenneth

Kenneth, I'll email you.