RethinkDB: here to stay or on the way out
2016-10-07 23:23 RethinkDB-in-or-out [permalink]
I'm worried and confused. The news these days is that RethinkDB is shutting down, at least the company. The database itself may live on. I had been attempting to write a really-thin Delphi wrapper for RethinkDB a number of times before, but got diverted into writing DelphiProtocolBuffer first, because it was used for the base of the server protocol and I couldn't find the Delphi support I was looking for (I'm sorry, but I'm a bit picky). Later it looked like they were planning on dropping this in favor of something else, but I wasn't able to investigate further at that time. In the mean time, I did a CouchDB, PostgreSQL and a MariaDB/MySQL wrapper, and a neat way to tie them together, and kept TMongoWire working, which turns out to be the one in the series of DB connectors that has the most success to date. (And which envolved learning all about PBKDF2...)
So now I'm confused about wether to keep RethinkDB on my to-do list? Or drop it? Or move it up to have a look sooner... Just like MongoDB, it may be that I won't really have a full-grown project anytime soon that is based on RethinkDB, but demand for a thin Delphi wrapper is out there and makes it worth it, I don't know. (Please let me know.)
And I'm worried about this new tale of an open-source-but-also-corporate endeavour going sour. It's only natural that when the projected sales don't appear to manifest, a project is best to get ended cleanly (as I recently witnessed over at the day-job). So I'm rooting for the open-source project to find a good new home. That the eco-system of volunteers may survive the orphanization. That the real-world users may carry enough weight to force the project into a viable after-life.
It's important to have a good climate to have entrepreneurs get enticed to take the step and start a company, but with this series of high-profile undertakings going bust, and aqui-hires where users get left in the dark, it starting to look like it's running out of control. Some people may called it a bubble, but I find the image of a pendulum swinging more appropriate. Is it swinging back already? I'm in no position to tell accurately. But I am worried.
For the record, I have had no economic gain from my open-source undertakings what-so-ever (up till now?) though OpenHub estimates about half a million went into xxm up till now. Not bad for a hobby...
But about RethinkDB, I guess I'll have to wait a little until the dust settles down...
xxHash: an extremely fast non-cryptographic hash algorithm
2016-09-23 23:50 xxHash [permalink]
I'm slowly but surely (finally!) working towards a (long overdue!) rewrite of DirDiff, but this time using threads and perhaps a different algorithm. Apparently there's still progress to make there, and I read good things about the diff git is using internally. Looking into that, some performance can be gained by doing the actual comparing on hashes of the data, instead of all of the data itself. Which lead me to xxHash, which should hit the sweet spot between fast enough and safe enough against collisions. (Unless I misunderstood.) I'm not sure if anyone thought of combining those two. But I may be looking into that for DirDiff 2... But since I didn't find a Delphi implementation rightaway, here it is, and it fits nicely with this collection of other hashes I did before. (Though xxHash is specifically a non-cryptographic hash...)
Delphi and MySQL or MariaDB
2016-09-15 21:12 libmysql [permalink]
Tadaa! I added a wrapper around libmysql.dll (to connect with MySQL or MariaDB) to my collection of really (really!) thin wrappers around things like ADO, or LibPQ (to connect with PostgreSQL) or now libmysql.dll. I also had the idea to align them as much as possible into almost the same interface. It's specifically not my point to have them be exactly the same, but very much almost so, just in case when I use this for a project and need to switch database back-end later on, the work that goes into that is minimalised and can concentrate around the different in SQL dialects. (for example the postgres branch here)
more unsolicited advice
2016-09-15 14:49 rant [permalink]
Make it look easy. Don't let any of the hard work you put in show. Hard work is what you put in onbeforehand, or need to afterwards just to keep up. So start early. Incredibly early, just because then you can still hide your beginners mistakes. Don't believe the crap the others are selling. They don't know. People that know won't just tell you. You could ask them, but in general they won't feel like telling you. Unless you can show you're determined to make it work. Resolve is currency, it buys you stuff. Just keep sure you buy the right stuff from the right people. People tell you to commit, but sometimes you need to let go, especially when it's clear it's not right for you. Be sure to break cleanly though, if you part rough it might cause more trouble than it's worth. Sleep. Right before you get tired the part of the brain shuts down that picks up what's going wrong because you've been up for too long. Wrap up slowly, calm down, drink, eat, wash, then sleep, enough. There is no sleeping when you're dead. Anything looks much better with a fresh head. Don't fear change. Change make a difference, and that holds people down. Just learn about what is going to happen, whether you make a change or not. Use that to keep control, to stay ahead of things. This one is hard: don't get angry. If you get mad it's because it's not clear why it's wrong. Find that out first. Check if you're even sure what is it at all. Then instead of getting livid, attack the misunderstandings. There are always misunderstandings. Even perhaps with you yourself, be sure to check. That's about it. One more thing, tough: have fun. Not always, some things are really tough. But as much as possible: have fun.
AES, DES (TripleDES)
2016-09-09 23:08 aesdes [permalink]
I've added AES and DES. DES may be deprecated, and Triple-DES may be soon, in any way it's cryptographically superceded by other ciphers, but still in use by some systems. I should do some work extra and change
array[0..7] of byte into
record l,r:cardinal; end; or even
int64 but I needed it only for something small and this works, so I'll leave it at that for now.
The order is an order is an order.
2016-09-02 15:19 anorder [permalink]
Then there was the day I decided to redesign. I had a hard time explaining. Not only why I thought a redesign was needed, but what is was exactly that I would do differently in this new design.
There would be a small distance between the data of the orders and the rest. The design would center around the idea that an order is exactly just and only that, an order. A request from a customer to have us do something. The distince between order-data and the rest wouldn't be large, it would more be something like a conceptual demarkation. In practice the order-data would be insert-only. A new order is registered, and after that read-only. This is easily explained to you, but caused shockwaves over the entrenched toilers of the company.
So to get closer to the common root of a great number of small problems, I started a design with just this, drawing up a model of reality where this is the first fact to note: the fact that an order comes in. When, by whom and for what. Not that the order is accepted. Not if it is payed for or not. Not if it's covered by customer credit or not. Not that it's cancelled or completed. Register different facts in different tables. Back to the books. Back to the 16th century penwright toiling in candlelit accounting books.
I sometimes joked that serious companies would require a stamp and signature to transfer this or that between departments. And silently I hope some still do. And even if none are left, it's up to us to see the romance in it. It's man-made structure to guide a man-made process, but it's structure and process none the less. Things break down swiftly when those go.
2016-07-08 22:52 jsonv103 [permalink]
jsonV now shows more sensible type labels, and only the fout-digit-hex-internal-value only on the most exotic variant types.
Ctrl+Shift+C now copies the value only, not just the node caption.
I also discovered a blocking issue when jsonDoc was trying to construct an array of int64 values. (more about that here)
2016-06-16 17:40 jsonv102 [permalink]
jsonV now accepts json-files that contain an array at the root level.
I also changed jsonDoc so all OleVariant arguments are passed const. This may break older versions, but should offer a little improvement in performance.
TIL: JVM and CIL both use the stack instead of registers.
2016-05-30 16:03 jvmcilstack [permalink]
Thing I learned today: both JVM and CIL use the stack instead of registers!
Sounds like a bad decision, not only because most other intermediate stuff I've seen uses SSA, but especially it makes the work for JIT-compilers that much harder.
At least, that's my first impression, just from thinking about what a JIT-compiler would have to do: assigning the processor's available registers to what they'll be used for in a section of logic, is a science and an artform all on its own; but if you don't have anything roughly materialized to work with from the beginning, you're bound to eke it out of the given logic and see what it's supposed to do, by following the things that hop on and off the stack... With the lot of guesswork, statistics and heuristics that comes with it.
I guess that's where the recent hype in JIT-compiler innovation comes from. If you get a cleaner conversion to something as if it was written with SSA, you'll probably get better results, but it still feels like a step backward and two steps forward.
And all this because I was comtemplating which back-end to (re)use in case I would have a go at having a play language of my own compile into something executable.
So, sorry guys, but I guess I'm back to looking into LLVM or GCC for a back-end... (unless...)
Just another debugging story from the trenches
2016-05-27 23:23 a-debug-story [permalink]
It's all logic, really. Once you get to the bottom of it. But getting there... Phew!
So I was working on something that had a line like this:
function TPostgresConnection.Insert(const TableName: UTF8String;
const Values: array of OleVariant; const PKFieldName: UTF8String=''): int64;
I forgot what went wrong in the beginning (it was somewhere yesterday between midnight and bedtime...) but to check something out, to debug without a debugger so to speak, I wanted to dump a string. Deep within the data-layer you can't really access the things that handle the output to the user, so I just do this:
And then run the application again and reproduce the issue you're tending to. But it doesn't throw this exception, it throws a quite severe access violation instead, strange. Not the error it started with, but none the less something strange going on in the exact same bit of code you're working on. But it was running late and I felt ready to go to sleep, so that is where I left it. It's always nicer to leave a project in working order at the end if the working day, but problems like this typically look different when viewed with a freshly rested set of eyes.
So the next day, let's move the debug exception somewhere higher up in the function body. Actually, all that happens before is handing the values in the
Values array... So instead, I put this in near the top:
for i:=0 to l-1 do sql1:=sql1+','+IntToHex(VarType(Values[i]),4);
Which gives me:
Hmm, so it looks like the list of VarType's of the Values array, but these values are totally bogus! The VarType constants are (at least in older Delphi versions) right there at the top of System.pas (the unit that's always included, even if you don't list in a
uses-clause), $0000 is varEmpty, but $FD18 is impossible since only
varArray = $2000 and
varByRef = $4000 are used of the highest 4 bits... And I'm sure I didn't pass any empties in. What is this?! Time to break out the real debugger then, I guess.
An xxm project pre-compiles files that have HTML and server-side pascal in the same source-files, into a full-fledged set of .pas files and a .dpr file, have that compiled, and then hot-swaps this new DLL in place to handle the next web-requests. So the usual development on an xxm project doesn't involve the Delphi IDE (I actually use atom.io since a while now.) but since the source is right there, you can open it in the plain old Delphi IDE and have the debugger start the DLL using something like xxmHttp.exe (the xxm handler that doesn't auto-recompile like xxmHttpDev.exe, since that would interfere with debugging.)
So, let's see what the debugger makes of this Values array the moment of this debug execption:
Values: (Unassigned, Variant array of Unknown, Unknown type: 21, Variant array of Unknown, Variant array of Unknown, Unassigned, Delphi exception EAccessViolation at $...
Yikes. But good news since with debugger it is doing the exact same thing wrong. That's something you can't take for granted, but that's another story.
So setting a breakpoint at this specific call to Insert, to see if the values get passed in:
Hey! Those are the correct values. But just one press of the F7 key later:
Bam, they're gone. Hmm, switching on the compiler "Use Debug DCUs" setting and retrying this doesn't bring me into any underlying code (sometimes some behind-the-scenes work is done on variants and strings for you), so nothing there...
And then it dawned on me! I know of one thing that can 'eat' your memory like that and it's compiler optimization.
By putting in that 'debug raise', using only an unrelated string, the optimization might conclude it can get away with releasing the memory of that Values array sooner, apparently disregarding the work that's been done on it. (Newer versions of Delphi may handle this differently, but that's another story...) So having a quick try with this:
for i:=0 to l-1 do sql1:=sql1+','+IntToHex(VarType(Values[i]),4);
if l<1000 then
And it works! So that's the weird misbehaving variant array out of the way. Now I feel extra silly not remembering what was originally the thing that I thought went wrong, was I just getting tired? This function is rather new, so I was just trying to debug for the first time, I guess. Still, makes for a nice story about debugging deeper and deeper until hitting the system, then finding out what's causing your trouble, calling on more of your knowledge of that system than usual.
(In case you want to know more about what I was working on in the first place, look here.)
Microsoft is Coca-Cola.
2016-05-23 10:10 mscc [permalink]
Or, at least, that was what popped into my head when I thought (again) about Microsoft open-sourcing the Windows operating system. Why wouldn't they? Coca-cola gave the super-secret recipe away at some point. It takes a certain stability, and vision, and momentum, to do that, but in my humble opinion both Coca-Cola and Microsoft have that.
Let's see how this could work. It's not because Microsoft would open-source Windows that they should stop selling it. Far from it. You can still buy Coca-cola, right? Have you ever bought cola made from the official Coca-Cola recipe, but concocted by someone else? Would you? Same goes for Windows. If you're in the market for a new computer, and want to run Windows on it, you'll probably go to the source, buy Microsoft, and be sure to get updates (and some free OneDrive space, and an Office365 trial...)
So I really am hopeful. Recently Microsoft has really (really!) opened up quite a bit, and even chose the MIT-license for some things, which was unthinkable just a few years ago. So a logic step would be to go all the way, and release the cash-cows, such as Windows, perhaps SQL Server... Even only perhaps there's a small chance they won't be the cash-cows for much longer... Computer sales is under pressure from smartphones and other hand-held devices. The database landscape is still suffering after-shocks from the NoSQL phenomenon, and from things like PostgreSQL and MariaDB, we roughly know what it takes to run a database anyway. So to ensure cash flowing in in the long run, it's almost a must to open up on old secrets. At least in my humble opinion.
Delphi and PostgreSQL
2016-05-13 00:19 LibPQData [permalink]
Searching for an alternative way to use PostgreSQL from Delphi? In the spirit of the other as-light-as-it-gets database connector wrappers, I've converted the header(s) to the LibPQ dll. See the LibPQ and LibPQData units in this repository:
To start an attempt to move to something more like a generic database layer, I've started this DataLank idea, but it needs more work. Actually I don't intend to develop it all the way into a full-fledged data-layer. Specialities of each specific database solution are so diverse and important, that I really don't want to hide them between an opaque abstraction layer. I want a database connector wrapper to be as light as possible, but also to allow access to the underlying technology. The main point for using DataLank, is to have less work changing DB's by using TDataConnection and TQueryResult types, but not no work. Adapting the used SQL to a different dialect will probably involve updating many (or all) calls the database anyway. But I'll post about that later when I put some more time in. (And perhaps write a libmysql.dll wrapper first...)
postgreSQL: date and time manipulation
2016-05-13 00:06 pg1 [permalink]
Babysteps! I've had the chance to dabble a bit in PostgreSQL, and after (years of) T-SQL, and some MySQL/MariaDB and SQLite, it's yet another SQL dialect to get a hang of. First impressions are good. It feels like a mature dialect. Searching the docs usually helps me find swiftly what I'm looking for. (And there's this ofcourse.) Porting an existing project makes you bump into the differences:
'f'instead of 1 and 0)
create procedure, but there's
create or replace function ... returns void
exec, but since it's all functions use
ID int identity(1,1) not null, use
ID serial primary key not nullor hook up a sequence by yourself...
insert into x (...) values (...) returning ID into NewID
I'll probably hit a lot more like these in the time to come, but that you should use
timestamptz) instead of
datetime2) deserves a separate mention. Especially there's no
extract, that's pretty straight-forward, but to replace
dateadd, you do something like it feels it should always have been: plain arithmetic.
PostgreSQL is pretty intelligent about operators (just do
select * from pg_operator, wow!) so just subtract two timestamps to get something of type
interval, something like
x + interval '2 days' result into exactly what you'd expect, and for constructing, apparently casting is pretty smart, for example:
select now()+(X.X||'day')::interval from (values (0),(1),(2),(3),(4)) X(X)
One more: instead of
datediff(dd,x,y)=0 just do this:
x::date=y::date pretty obvious if you think about it. Like I said, PostgreSQL made a pretty good first impression.
2016-05-03 19:17 plasm [permalink]
It is also a play thing to try and tame this beast that is assembler. Not any assembler in particular, but just in general what it is to try and make things work with a limited set of registers and instructions and having to shuffle anything that doesn't fit in those to and from memory. If I put more work into this, it could go two ways:
I can evolve the machine code further and try to design a binary encoding for the possible instructions, and have the code turn up in the memory as it is accessed by the normal registers. (It is not now, the
I register now indexes into a separate set of instructions, possibly with overlapping values.)
Otherways I could try and build a compiler that compiles code into this set of machine instructions (perhaps based on this one), finding out along the way what is all the fuss about register allocation and heuristic optimizations...
And ofcourse the both meet and I could try to build a compiler straight into these virtual machine instructions, but I have the luxury of not having to do this. And since it's all about learning and exploring by doing, it'll probably get me stuck faster.
But for now I think I'll let this rest for a little while. See how it looks when I come back to it after a while.
See also https://github.com/stijnsanders/plasm
2016-05-03 19:02 xxm430 [permalink]
- xxmISAPI: fixed issue with open connections not accepting new requests
- revised context object lifetime management (lock-less reference counting)
- demo 7 "Resources": added "If-Modified-Since"
- demo 12 "Long Polling": added example EventSource (text/event-stream)
SQLite: "collate nocase"
2016-03-11 20:35 sqlitenocase [permalink]
Learn something new each day! (Or also: I should have known about this!) SQLite is so strict! But in a good way. It's in the little things that come up now and then that you notice how delightfully strict it is designed, with reliability and safety up front. And no extra's. So it's quite logic that sorting a column with string values* is sorted case-sensitive, with al upper-case letters before the lower-case letters. To change this, add
collate nocase either to the
create table statement or the
order by clause of queries. I've picked up somewhere that it's better for performance to have it in the
create table statement, but I haven't checked this with the documentation or by testing in a test setup (yet), but for now I just went for updating the tables since changing all the queries would take much more work. That said, updating: not quite.
Alter table doesn't have
alter column apparently, but it does have a
rename to, and
insert into TblUpdate select * from Tbl works great, even with an autoincrement primary key (if you fields are still in the same order), so it's a relatively painless transition.
And this barely a week after I find out about the busy timeout!
(*: due to manifest typing different records may hold values of different type in the same field, but in practice you'll usually follow the type as determined by the schema.)
"What will you do?"
2016-03-11 11:50 whattodo [permalink]
"I'm trying to decide between making a big change for a small number of people, or making a small change for a big number of people.
What I dream of is making a big change for a big number of people, but that's everybody's childhood dream, isn't it?
Then again, most people that did make a big change for a big number of people, were only trying to make a small change for themselves."
2016-03-08 10:27 20160308i [permalink]
Nu moest ik toch even lachen met deze: N-VA: "Turkije hoort niet thuis in EU. Nooit" Zou mijn voorspelling van vijf jaar terug dan alsnog uitkomen? Dat ze Europees gezien het zichzelf onmogelijk gaan maken?
De laatste dagen vroeg ik me al af wat er gebeurd was dat ze in Brussel plots dikke vriendjes spelen, en zelfs niet-Erdohan laten zeggen dat ze er echt wel bij willen. Ah, een gigantische stroom oorlogsvluchtelingen natuurlijk.
Even terzijde, jammer dat ze in het nieuws er niet in slagen een juist beeld te geven van de juiste mix Syriërs-Irakezen-Afganen-Afrikanen-etc, maar dat is dan ook een moeilijk geval, zelfs voor de betrokken diensten ter plaatse.
Maar Turkije dus, en de EU, ik ben aan het aftellen tot wanneer het woord Cyprus eindelijk valt. Waarschijnlijk na ze iets in de vorm van een akkoord hebben over wat ze voor (niet met!) de vluchtelingen gaan doen. En heel misschien valt zelfs eerst nog het woord Koerdistan. Soit, ik hou nog even deze goed in de gaten...
jsonV: plain and simple JSON viewer based on jsonDoc
2016-02-19 21:39 jsonV [permalink]
I decided to create a really simple JSON viewer based on a TTreeView instance and jsonDoc, which itself is heavily based on OleVariants (I really hate long lists of overloads!) so I have some VarType values handled, but show the rest
'('+IntToHex(VarType(v),4)+')'. It could do so much more, but it then would be no longer a really simple JSON viewer. (I currently use Atom with beautify for JSON editing.) I've commited the source in the jsonDoc repository, see here for a ready-to-run executable (It's an exe in a zip, so some browsers and virus-scanners may cry havok about the security risk):
just an idea: var in code for variables of limited scope
2016-02-03 16:18 var-limited-scope [permalink]
I've been reading around a lot about the new programming languages (Go, Rust, D, WebJavaECMACoffee6asm.ScriptX) and there's ofcourse this thing, but I still do most work for work and distraction in Delphi. So all of a sudden, in a large code block (I know, I know, one is supposed to keep their methods many and small) I thought of something like this:
more_code_that_makes_the_world_tick(); //perhaps even using 's' but the one declared way above...
In light of the asynchronous programming most languages get pushed upon them, and the need to have 'scope' outliving the usual runtime of the function call, this may have been considered. But purely on the basis of syntax, this could be done. A 'var' token at that point is never valid, so could get used by a parser to initiate a declaration block, followed by a begin/end block, that the compiler would know how to convert into a separate scope (with unwinding, and auto-calling release on interfaces/reference-counted-instances)...
But it currently doesn't. The closest thing you could get is define an anonymous function and call it once there and then, but that may look ugly and is too much code for what it does, in my humble opinion.
So for now I will just be (re-)using a single
s:string; on top, I'm afraid. (I feel your sqeam, but I'll be really carefull I promise.)
Google Finance: BTCEUR
2015-12-11 08:22 gfBTCEUR [permalink]
Hee, sinds wanneer hebben ze deze toegevoegd?
2015-12-11 00:00 metaclick466 [permalink]
- when suspended, time down to unsuspend only on self
- time down on buttons on self is always a normal click
- setting: hide unless mouse over (only show corners)
- setting: alpha blend level
2015-12-10 23:56 metakeys446 [permalink]
- issue changing font with "hide unless mouse over"
- replaced "Wingdings 3" for "Symbol" in default kbl's
- code 666 closes MetaClick
- code 777 opens settings dialog
- codes 1000..1999 loads "000.kbl".."999.kbl"
- code XXXYYY resizes to XXX by YYY pixels
- code KKKXXXYYY loads "KKK.kbl" and resizes to XXX by YYY pixels
- hide unless mouse over (only show corners)
CouchDB connector: TCouchDBConnection
2015-12-03 07:43 TCouchDB [permalink]
After creating my own wrapper around ADO with the least possible of extra's, but with the most ease-of-use I could get; I tried the same for mongoDB; then Redis; then RethinkDB, but first had to tackle protocol buffers, but in the mean time both advanced to a new version and RethinkDB dropped protocol buffers, so I dropped my RethinkDB connector and put both on the to-do list again.
So now I thought I'd look into CouchDB, it's all HTTP so it should be easy enough. The point is you connect to your CouchDB instance on an address and port that's not publicly accessible, so a full-fledged HTTP library would be too much (though it's still highly recommended nowadays to use TLS over internal connections as well, but that's on the roadmap for xxmHttp). For handling JSON I can re-use what I created for TMongoWire: jsonDoc. So there it is. For now it offers a basic call to exchange JSON with the server, but this should be enough to unlock the main features CouchDB has to offer.
Types of Information
2015-11-24 11:04 typesofinformation [permalink]
Life would be good if all we had was information. If only we could get all the information. We can't handle that much information, so we build systems to handle the information for us. Strange things happen when information comes in. Good systems are designed to handle these well. In designing information processing systems, you have to cater for the following.
Some input signals contain no information. They are either damaged in transport, incomplete, or not of a correct form for the system to handle. Either report them to find out if repair is possible, or keep count of them to be able to report about the health of the system.
Some input signals come in twice, and contain the same information. Or do they? If possible try to have the last step in the chain report if an event did take place twice, or if it's an echo on the line. Sometimes a clerk does drop a pack of cards and enters them again just to be sure. Be ready to take only the new cards.
Some information is wrong. Us humans do make mistakes. The system sometimes doesn't know. It processes a signal, of the correct form, holding valid information. Then again, the fact that some information already in the system may be incorrect is also information.
Don't expect to get all information. There is always more.
More input is coming in. Sometimes we know how much input is still waiting to come in. We'll roughly know how much information there'll be added. In most cases it follows a measurable trend. In some cases it follows the business.
Information processing is one thing, but does it deliver the required new information? Is there more to mine out of the amassment? Sometimes the numbers can show what you need to know, but do you know where to look?
It's hard to design for things we don't know we don't know (yet). But it serves to be prepared. New things have a knack for looking a lot like something we have already. Sometimes they deserve a new module, sometime just a new category, but don't forget to put the existing items in a category also.