New scripting language

[CyberRascal]

So it has come to this.

The LuaInterface library seems to be prone to crashing. Just as importantly, there is no way for FreePIE to prevent these crashes. The latest issue are buffer overruns smashing something deep in CLR land, and there just is no recovering from that. That I know of.

Thus, I've been looking at alternatives. First I looked for another, fully managed Lua library, but support doesn't seem overwhelming. After that I considered writing a Lua lexer and parser and transform Lua -> C#, and dynamically JIT it into an assembly. I like this idea, any generated code lives 100% inside the CLR, better error handling support (native exception support for one), and probably a healthy speedup to boot.

(you can skip ahead till after the code example if you want - you'll only miss a few technical deliberations)

While it would be fun to develop, there are many kinks to workout. Most of them could probably be worked out, but there is bound to be some impedance mismatch between Lua and C#, not the least of which is that Lua is dynamic and C# is mostly not.

CyberVillain flippantly asked 'why go to all that trouble when you could just use C# directly?'. It's got some merit, no impedance mismatch, no transformation, no parsing. Just some minor hassle with making the user experience a REPL-ish interface.

That's when I found Roslyn. It's the microsoft-implemented replacement for the current C# and VB.Net compilers. It is complete with syntax analysis (basically giving us intellisense for free), compilers and most importantly: a ready-to-go scripting engine.

I've fiddled around a bit with it, it's actually in a local feature branch of FreePIE, and it's up and running. Very easy, but this was just a spike to figure out if it's feasible or not. To give you a taste of the syntax (this works right now):

Code: Select all

if(starting)
	trackIR.Update += (s, args) => diagnostics.watch(trackIR.Yaw);

Now, what I want input on is this... What do you think of C# as the FreePIE language of choice? Do you have any other suggestions? What do you think users that might not be experienced programmers might think of the syntax vs that of Lua?

In the near future Roslyn will provide support for the dynamic keyword and everything that entails - which means that the syntax can be even more script-like:

Code: Select all

x = 10;
y = x + "hello";

Notes:

Roslyn is .net 4.5. That version of .net doesn't run on XP. How many out there are using XP? Can we drop support for it?

Latest update:

Roslyn and C# is scrapped as scripting language. Python is now implemented however, and the code is for now available at https://github.com/maxmalmgren/FreePIE/

[CyberVillain]

I like the idea, but C# is pretty far from Lua, VB is still not a dynamic language, but its more "script like". I have no problem with C# but lets see what the community says. Will try your POC tonight and get back to you, are all plugins working or is it hardcoded to trackIR?

[CyberRascal]

All plugins are working (only tested TrackIR though).

Another problem that you, CyberVillain, just discovered (shame on you for using XP) is that .Net 4.5 is not supported on XP. Roslyn is right now .Net 4.5 only (will probably always be so).

This means that FreePIE would have to drop XP support. How many users actually use XP as of Q3 2012? Can we drop support? Is anyone here using XP?

[CyberVillain]

(shame on you for using XP)

Sadly I can not make my customer upgrade their aging IT platform :/

[brantlew]

@CyberRascal: Thanks for the analysis. Too bad about this because LUA has been good to us so far.

I think it would be a bad idea to drop support for XP. It still accounts for something like 40% of all Windows installs in the world so that seems very significant to me.

As far as C# goes, do we really want non-programmers to have to understand scoping, type rules, class definitions, and C syntax? And of course C# in it's most esoteric meta-form (=>, LINQ) is ridiculously obtuse to read - both for the average guy as well as seasoned C programmers. And if we're making it a requirement to understand C#, then why not just provide an API instead and tell them to compile their own script?

No, I've always thought that a very simple script-like language is an important goal. Especially if we are trying to wrestle mind-share from GlovePIE which has a very simple syntax. Maybe if we have a simple custom pre-processor to "compile" a simple script into a more complex language like C#. 98% of all scripts are in the form of simple assignments (=), function calls, and "if" conditionals so we don't need something terribly flexible. Does it really matter if we don't have a do/while loop or a foreach construct? No.

[mahler]

Hmm, it might be tempting to go for something new and shiny with the promise of being crash-prone.

Roslyn seems like a good option on paper, but it's relatively new and I agree with brantlew that giving up XP support just isn't realistic at the moment (although I do think that 40% does not represent the FreePIE userbase)

A short term more pragmatic approach could be including good example Lua-scripts in the distribution of FreePIE and creating user-awareness about possible ways to avoid crashes.

[CyberVillain]

Hmm, it might be tempting to go for something new and shiny with the promise of being crash-prone.
But I doubt that it will solve crashes all together. I'm sure with Roslyn it's easy to break the system in another way.

Roslyn seems like a good option on paper, but it's relatively new and I agree with brantlew that giving up XP support just isn't realistic at the moment (although I do think that 40% does not represent the FreePIE userbase)

A short term more pragmatic approach could be including good example Lua-scripts in the distribution of FreePIE and creating user-awareness about possible ways to avoid crashes.

There is no way of avoiding crashes :/ Just minimize the risk, but in the end its just about fluke luck

[mahler]

.... But I doubt that it will solve crashes all together. I'm sure with Roslyn it's easy to break the system in another way.

There is no way of avoiding crashes :/ Just minimize the risk, but in the end its just about fluke luck

I had just removed that part of my comment, because it was pure speculation. I shouldn't have said that without knowing more about Roslyn. I have no idea how they manage the code. And I'm looking at the website: http://msdn.microsoft.com/en-gb/roslyn but I can't read any of the docx-documents because I'm here at work on a Windows XP machine with an older version of Word. Oh the irony.

[CyberVillain]

There is no way of avoiding crashes :/ Just minimize the risk, but in the end its just about fluke luck

Just to clarify, This was aimed at Lua, not Roslyn.

edit: What I meant was That there is no way from script or .NET code avoid crashes with the current Lua engine, its just a matter of fluke luck. For example Brantlews console program hasnet solved the problem, it has just minimized the risk since System.Console uses less resoruces than WPF and its less likely that Lua overwrites important CLR memory

[CyberVillain]

And of course C# in it's most esoteric meta-form (=>, LINQ) is ridiculously obtuse to read

Hold your horses, so you favor

Code: Select all

foreach(var item in items) {
   if(item.expression)
      item.DoSomething();
}

Over

Code: Select all

items
   .Where(item => item.expression)
   .ForEach(item => item.DoSomething());

??

[mahler]

There is no way of avoiding crashes :/ Just minimize the risk, but in the end its just about fluke luck

Just to clarify, This was aimed at Lua, not Roslyn.

edit: What I meant was That there is no way from script or .NET code avoid crashes with the current Lua engine, its just a matter of fluke luck. For example Brantlews console program hasnet solved the problem, it has just minimized the risk since System.Console uses less resoruces than WPF and its less likely that Lua overwrites important CLR memory

Ah, ok.
However I just found your thread on stackexchange: http://stackoverflow.com/questions/1035 ... inite-loop
And there seems to be an answer from last month about sandboxing Lua.
You probably already saw that, but I'm curious why that didn't work...

[CyberRascal]

@CyberRascal: Thanks for the analysis. Too bad about this because LUA has been good to us so far.

I think it would be a bad idea to drop support for XP. It still accounts for something like 40% of all Windows installs in the world so that seems very significant to me.

As far as C# goes, do we really want non-programmers to have to understand scoping, type rules, class definitions, and C syntax? And of course C# in it's most esoteric meta-form (=>, LINQ) is ridiculously obtuse to read - both for the average guy as well as seasoned C programmers. And if we're making it a requirement to understand C#, then why not just provide an API instead and tell them to compile their own script?

To be fair, at the basic level the Lua syntax is pretty close to C#, when you get down to function level that is. Check out the update listener in my first post. The Lua flavor supported in FreePIE right now would express that as:

Code: Select all

if(starting)
then
	trackIR.Update:Add(function()
		diagnostics:watch(trackIR.Yaw)
	end)
end

To write that yourself, you still need to basically understand that you are adding an event listener to an event and that you must only do so when "starting" is true. I'm obviously biased here, but in my opinion at that point the basic C# syntax shouldn't be that much of a hinder. You don't need to pull data out of a database, transform it with IEnumerable.Select and aggregate it using IEnumerable.Aggregate.

Another entirely valid script is:

Code: Select all

Console.WriteLine(trackIR.Yaw);

Also, there is no reason to know anything about classes and methods. All the snippets I have written in this topic has been complete scripts, written in the FreePIE editor and executed as is - no classes, no definitions, nothing. Just so that is clear.

Your (brantlew) paragraphs about both a custom preprocessor and class definitions etc indicates that you thought that a user of FreePIE would need to write a lot of plumbing code to execute the scripts above. Is that true? Are you still as opposed to the actual examples I wrote?

I will look around a bit concerning Roslyn. It seems it could be possible to run it in .Net 4, and in that case, that issue is resolved at least.

Anyway, thanks for the input! Keep it coming I just want to make FreePIE as stable as it should be, not really push for C#.

mahler: That is pure irony The problem with LuaInterface seems to be the marshalling of C# callbacks to lua code. In short, whatever code you execute has the potential of crashing FreePIE - it's just a matter of whether or not the memory that is overwritten is vital or not. The crashes we are seeing are probably caused by a LUA function (native function atleast) overunning some heap infrastructure and thus causes the crash. It's like having a mine in the middle of a street - most of the time, it doesn't explode, but it's still there.

[CyberVillain]

@Mahler CyberRascals response to you is pretty spot on why I didnt do that, sandboxing Lua into a seperate process does not solve the problem is just builds around it, it can still crash due to overwritten memory

[CyberRascal]

Also, note that the TrackIR plugin / module actually uses that kind of out-of-process execution. Since it uses a dynamically loaded native dll from an unknown third party (Natural Point, FreeTrack, FreePIE etc) it's impossible to know if it handles memory correct etc. Thats why I opted for loading it in its own process.

The amount of effort to get IPC up and running is however MUCH greater when we're talking about the whole scripting engine.

[brantlew]

And of course C# in it's most esoteric meta-form (=>, LINQ) is ridiculously obtuse to read

Hold your horses, so you favor

Code: Select all

foreach(var item in items) {
   if(item.expression)
      item.DoSomtehing();
}

Over

Code: Select all

items
   .Where(item => item.expression)
   .ForEach(item => item.DoSomtehing());

??

Absolutely. The top one actually reads like more like natural language (even more so if the language include the "then" keyword). The bottom one is mathematical symbology. The top one can be taught and remembered much easier. I feel the same way about the cryptic "?:" operator in C. It doesn't add any more expressive power and only serves to make the code harder to read.

But to tell you the truth...as an engineer I actually prefer this syntax, because all the operations are explicit and can be debugged it at every level.

Code: Select all

for (int i=0; i<Items.length; i++) {
   Item a = (Item)Items[i];
   if (Expression(a)) {
      DoSomething(a);
   }
}

For example, if Items has 10,000 elements in it and it's crashing on element 8,567, I can just put a conditional breakpoint on i. Meta-programming resists this type of low-level debugging, and I always find it to be counter-productive once bugs start popping up.

[CyberVillain]

But to tell you the truth...as an engineer I actually prefer this syntax, because all the operations are explicit and can be debugged it at every level.

You can debug a Linq expression too at every level, but lets not get off topic I recall we had this discussion some time ago so its already established that we have differences when it comes to coding standards

[brantlew]

@CyberVillain: Agreed. Let's not have a holy war about meta-programming, because we clearly live on opposite sides of the universe in this regard.

[mahler]

mahler: That is pure irony The problem with LuaInterface seems to be the marshalling of C# callbacks to lua code. In short, whatever code you execute has the potential of crashing FreePIE - it's just a matter of whether or not the memory that is overwritten is vital or not. The crashes we are seeing are probably caused by a LUA function (native function atleast) overunning some heap infrastructure and thus causes the crash. It's like having a mine in the middle of a street - most of the time, it doesn't explode, but it's still there.

Thanks for the explanation. So, basically the problem is with LuaInterface and that's why you said ".... looking for alternatives" in your opening post. I looked into some of the issues posted there by CyberVillain and indeed no reactions by the owner since 2011.

I don't like LINQ programming notations either because I'm not used to them, but to be honest, I don't really care that much about the way the scripts are created. I learnt Lua from the examples when I was trying to get FreePIE to work, I'm sure I could do that same with any other script with proper examples and documentation.

If you list the original arguments for choosing Lua ... and you can match these with Roslyn, then go for it. There are plenty of reasons why it's not perfect, but none of them apply to me.

[brantlew]

I think LUA was chosen merely as a convenience. It has a decent amount of mind-share and there was a lot of toolkit support. So it just sort of "plugged-in" without a lot of fuss and enabled us to bypass all the details of script parsing at the beginning. As it turned out - it also included some useful features and meshed well with C#, but syntactically it's not my favorite.

[CyberRascal]

brantlew: Which language do you think fits better? I'm not really well versed with many scripting languages, pretty much only Javascript and now Lua..

[brantlew]

I don't have a candidate. Actually, I liked the Lua interpreter just fine and it's great to have an engine take care of all the messy code logic details. I guess I always assumed script syntax was trivial enough to be parsed with custom code. I had considered creating a simple script compiler that just translated a basic script syntax into Lua code and fed that to the Lua interpreter. Too bad that engine is unstable. I don't know much about the C# engine that CyberVillain is talking about, but if it's true that you can just take uncompiled C# code and feed it into the .NET engine for execution, that would be great. Just run a script through a pre-parser to put all the syntactic sugar like scope braces {} and "var" statements in there and run it as native C#.

[CyberRascal]

The syntax is pretty much like this:

Code: Select all

var engine = new ScriptEngine();
var session = engine.CreateSession(context);
session.Execute(script);

So yes, it's very easy. The code is executed in a REPL manner.. The loop is something like:

Code: Select all

while(!stopped)
{
    session.Execute(script);
    foreach(var plugin in usedPlugins)
        plugin.DoYourStuff();
}

It gets a little more complicated since we've got threaded plugins, and I need to inject the globals into the context. Roslyn doesn't support the dynamic keyword yet. Once it does, you won't need to declare variables at all. We will need to use Roslyns syntax analysis to determine what variables should be persistant - at that point we can probably just find all unresolved variables in the syntax tree and declare variables with the type. Most of the work is done by Roslyn.

I think the most sensible thing to do is let everyone try it out. I'll fix up the tryout somewhat, then upload an executable. What do you think?

[brantlew]

Doesn't Roslyn preclude XP though? That's a big sacrifice - don't you think?

[CyberVillain]

Doesn't Roslyn preclude XP though? That's a big sacrifice - don't you think?

There is no excuse for a consumer to use XP these days

[brantlew]

So let me get this straight...you are telling the customer base (whom are largely non-programmers) that they must learn C# and that they cannot use Windows XP. All so that the developers will have an easier time creating the product.

It sounds backwards to me.

[CyberVillain]

So let me get this straight...you are telling the customer base (whom are largely non-programmers) that they must learn C# and that they cannot use Windows XP. All so that the developers will have an easier time creating the product.

It sounds backwards to me.

Lua x = trackIR.Yaw

C# var x = trackIR.Yaw;

I dont think most people have problems with the difference

I'm guessing most users of FreePIE are powerusers and they have upgraded from XP a long time ago. Do we really need to support that one %

edit: BUT, there is a problem, CyberRascal, how are we supossed to scope variables?

if(starting) then
x = 0
end

Does not work in c#

[CyberRascal]

Concerning point #1, learning C#:

You don't have to learn all of C# - you can skip class definitions, entrypoints, method definitions, scoping, among many others. Don't you agree that the above examples are relatively simple? Even the one using the lambda syntax can easily be taught, or memorized.

What we want to avoid is the plumbing code, like the things you actually can skip that I mentioned above.

Concerning point #2, dropping XP support:

I think the pros would need to be overwhelming in support of say Roslyn before we can accept that. But it's not just an issue of making lives easier for the devs. We need to use something solid - for example, right now code completion is very shaky. It's a naive implementation probably causing more confusion than not. Using Roslyn, we get a proper C# lexical analyser. Not to mention that we need something that doesn't crash. Since Roslyn is fully managed, atleast FreePIE can react to the issue instead of dropping dead.

To put the issue on it's edge: If we could use XP, is C# good enough?

[CyberRascal]

CyberVillain:

Once Roslyn supports dynamic objects, the following syntax should work:

Code: Select all

if(starting)
{
    x = 0;
}

Right now, I can solve it by pre-parsing the script (using Roslyn) and collect all variable definitions, and declare them as "globals". This means that the statement "var x = 0;" would only execute once. You don't need to put it in if(starting).

Syntax:

Code: Select all

var x = 0;

x++;

It's a little fuzzy that the declaration is only run once though.

[CyberRascal]

Another alternative we could use is Python. Anyone have any opinions on that?

IronPython is relatively well supported, I think...

[brantlew]

Another alternative we could use is Python. Anyone have any opinions on that?

IronPython is relatively well supported, I think...

What irony. I was JUST about to post the same thing. I don't know anything about it, but I think it would be a great idea to take a look at it.

[CyberRascal]

Me neither, but I do know that python uses whitespace for lexical scope / tokens. Or am I wrong? It seems like that could be a complexity even more difficult to understand to a non-programmer. What do you think?

[CyberVillain]

I used IronPyhon long time ago when I did my thesis work, It was really good and stable like a rock. I dont like the syntax, but I'm not a hater

@Cyber Yeah, its very prone to errors even for an experienced developer

edit: I dont remember which C# specific stuff it supports, we need Properties, methods, events and enums

[brantlew]

Me neither, but I do know that python uses whitespace for lexical scope / tokens. Or am I wrong? It seems like that could be a complexity even more difficult to understand to a non-programmer. What do you think?

You're right. Python scopes by indentation. My guess is that non-programmers would be more comfortable with it than programmers since it looks like document indentation. I can understand why many programmers are uncomfortable with the ambiguity of it. But considering how simple the rest of Python syntax is, the positives may outweigh the negative in this case. Plus, since FreePIE scripts tend to be very simple with only one or two scopes it might not be so bad.

[CyberRascal]

I definitely think it is neater than the then-end construct. I've made another branch, reusing much of the code I wrote for roslyn. It's up and running, was even easier than the Roslyn one.

It'll be available to download from my github (https://github.com/maxmalmgren/FreePIE/) in a little while. It's far from completed, but supports most stuff I guess... Taste of the syntax (tested in future pinball):

Code: Select all

if starting:
	x = 0

x = x + 0.01;

if x > 90:
	x = -90

trackIR.Yaw = x

[brantlew]

@CyberRascal: That's awesome !! Python syntax will definitely be an improvement over Lua - and hopefully the stability will improve as well. Great job!

[CyberVillain]

Nice work! Checking out the code as we speak

[CyberRascal]

Thanks for the support guys

Anyway, things are moving pretty fast with python. We are almost there, but we lack two major things - syntax highlighting and code completion. Hopefully we can get working syntax highlighting from somewhere soon, but code completion might be harder. I will look around a bit for a compatible lexer for python.

I think we should probably release a new binary before code completion... What do you guys think?

Latest code is on my github right now if anyone want to try it out.

[CyberRascal]

Also, mahler: What do you think about using Python?

Anyone else have any comments? I'd rather not force something like this onto anyone :p

[brantlew]

Thanks for the support guys
I think we should probably release a new binary before code completion... What do you guys think?

Absolutely. Engine stability is vastly more important than code completion.

[CyberVillain]

I agree, but cant you just use the old CC with the expression tree changed to Python syntax instead of Lua, and then we can support a more advanced lexer in the future. The Old CC was at least a good reference manual to the available plugins and helpers.

I tried to find a ready to go highlighting file yesterday, but the one i found was a old one and didnt work with latest AvalonEdit. Will look more if i find time between meetings today