Build queries for Lucene/Solr indexes in PHP 2

We use Solr a lot at InterNations. Beside usual full text searches, we use it every time we need to receive documents nearly free of charge. It is fast, it is stable and, after some wrestling, our data import works very well as well.

For more than a year we added more and more functionality to a component for building Solr/Lucene queries programmatically. We provide two different ways to create queries: for complex queries we use an expression builder, for simpler ones we have a string based class. A huge advantage of a programmatic API is security: while Lucene’s query language is read-only and therefore non-destructive, query injections can lead to serious data breaches, which both components help to avoid by escaping input strings.

We now feel they are mature and stable enough to be released to the public: say hello to internations/solr-query-component.

Filed on 23-03-2013, 21:09 under , , , , & two comments & no trackbacks

Batch-generating SSL certificates 3

I had an itch to scratch this day: generate a huge amount of SSL-certificates for a set of whitelabels. Every whitelabel has it’s own domain and needs it’s own SSL certificate. I don’t want to spend n ⋅ x ¤ to buy them just for testing purpose (generating a huge number would have been an issue too if I bought them but …). So I needed a script to do it. We had one before to generate a single one with the typical user interactions. While hacking it down I found out a few neat things about OpenSSL which I didn’t know before and which I think are worth sharing.

Disclaimer: the script is not hacked together with security in mind. It simply doesn’t matter for me in this specific case.

So after alot of chit-chatting, there we go:

if [ -z "$DOMAIN" ]; then
    echo "Usage: $(basename $0) <domain>"
    exit 11

Obvious: we need to call the script with a domain as a parameter. Next is a simple to fail when a certain condition matches:

fail_if_error() {
    [ $1 != 0 ] && {
        unset PASSPHRASE
        exit 10
export PASSPHRASE=$(head -c 128 /dev/random  | uuencode - | grep -v "^end" | tr "\n" "d")

We generate a passphrase by reading 128 byte from /dev/random, encoding it with base64 and joining the lines. A simple trick if you can’t rely on pwgen or makepasswd. Next is to define the subject string. But we’ll talk about that later:

C=<Country Code>
organizationalUnitName=<Unit Name>

Of all things generating a fresh RSA key comes first. The neat trick to avoid user interaction here is to use -passout option. This option takes an protocol argument similar to PHPs stream wrappers. env:<envvar> for environment variables, pass:<password> if you like to pass the string directly, file:<filename> to read from a file, fd:<number> to read from a file descriptor and stdin – well – to read from the standard input. Of course we fail on error.

openssl genrsa -des3 -out $DOMAIN.key -passout env:PASSPHRASE 2048
fail_if_error $?

Now we generate a certificate request (.csr). -batch simply supresses user interaction. And here comes the $subj variable into play which allows us to override our certificate requests defaults specified in our openssl.conf. Most of the time the OpenSSL configuration file should be located in /etc/ssl/openssl.cnf but if you are on Mac OS X it is located in /System/Library/OpenSSL/openssl.cnf. The subject variables you can override are defined in this configuration. It’s an INI file, just search for the section [ req ] and look at the keys distinguished_name and req_attributes. There values redirect to the section name we are searching for: here you can find the candidates you might want to override in your subject string. Just extend the $subj variable according to your needs. The $subj variable lines are joined with a slash as a seperator – this is how OpenSSL likes it.

openssl req \
    -new \
    -batch \
    -subj "$(echo -n "$subj" | tr "\n" "/")" \
    -key $DOMAIN.key \
    -out $DOMAIN.csr \
    -passin env:PASSPHRASE
fail_if_error $?
cp $DOMAIN.key $DOMAIN.key.org
fail_if_error $?

Strip the passphrase from our RSA-key to not get prompted when Apache (or any other webserver) starts:

openssl rsa -in $DOMAIN.key.org -out $DOMAIN.key -passin env:PASSPHRASE
fail_if_error $?

Last step, create the certificate file:

openssl x509 -req -days 365 -in $DOMAIN.csr -signkey $DOMAIN.key -out $DOMAIN.crt
fail_if_error $?

Now: deploy the .key and .crt file to your webserver and set up your configuration.

Filed on 14-08-2009, 22:10 under , , , , , & three comments & no trackbacks

Security "to go"? 5

I’m a huge fan of PHP-IDS. Mario Heiderich and Christian Matthies did an incredible job polishing this tool, adding new features and trying to catch every esoteric attack signature. However I have the feeling there is some confusion (german) about what intrusion detection is for. On a server, intrusion detection is used to diagnose a break in. First of all you do everything not to let your server go down. You have a firewall, you try not to expose services to the outside, you do SSH with port knocking, you put a risky service into jail or chroots, you use the Suhosin patchset and so on. There are various strategies how to harden a server. The hardening is the barrier against break-in attempts.
If the hell freezes, the intrusion detection mechanism comes into play to make sure the attempt is not overseen and the machine does not become yet another zombie in a bot net. PHP-IDS is an intrusion detection tool on the application level. Application firewalls know about a certain protocol and its structure (e.g. HTTP) and inspect the protocol to detect attack patterns. Some of them are even capable of learning from usual request signatures and enforcing rules based on the learned data. There are various commercial products to achieve application firewalling. PHP-IDS does the same for free and sits directly on the webserver in the scope of the application. For personal usage or projects with a lower budgets who can’t effort expensive products, it might be a good supplement. Beside being a supplement, application firewalls are a valid use when security becomes an urgent problem: a lot of heavily flawed software is designed (often it is not even designed) and developed without a developer even heard about security: “Yes you can inject HTML, that’s a feature!”, “‘ OR true/* lists you every item, isn’t that cool?”. If such projects become popular, application firewalls might be an option to hotfix the disaster. But nevertheless the application needs to be fixed.
The very immanent issue with application firewalls is that there is no other place to know exactly what’s proper incoming data for the application – except in the application itself. That’s why application firewalls can never be perfect. IDS is needed for the 2% the developer forgot. So it is not like coffee to go. It is like having the coffee and adding milk or sugar. Having milk without coffee seems pointless to me anyway.

Filed on 20-05-2008, 21:09 under , , , & five comments & one trackback

Security reporting the productive way 0

This morning I reported an XSS-vulnarability to mugshot. An hour later I got a response, that they are investigating the issue and deploying a fix. Six hours later the fix was deployed and I was notified again. This is exemplary!

Filed on 22-10-2007, 16:04 under , , & no comments & no trackbacks

Preventing cross site scripting by design 4

This is more or less a reply to Dynamic global functions in PHP. My main problems with delegating escaping in the template is the fact, that the people who normally work with templates, are frontend developers and designers. Those who do not and should not care about web security. That is a programmers/architects field. Nobody of the frontend developers should have the possibility to create security level artefacts by accident. So, when a value arrives the template, everything should be done. No special escape-calls should be necessary. I will show you how we do escaping and template value sanitizing at Neu.de. But let’s step through all common models in order to explain, why they are bad. I assume you know the basic MVC-terms, I will use here mostlye view and controller action. First of all, the most common approach. Just assigning variables as-is to the view component:

The second – much better approach – is to escape values before accessing them in the template. This is fine as long as you do not use objects in your templates.

If you have complex, nested objects encapsulating complex business rules, you do not want to convert them to an array to make it possible to escape them afterwards because of speed concerns. So if you pass an object with a method which returns fragile user input, your escaping logic is bypassed. See:

The solution is to wrap assign objects in mock objects. You can easily implement a mock object builder using PHP5s reflection features and create a simple proxy which escapes the return values of every call – or – if an object is returned – wrappes this return object in another mock object. And so on and so on:

Once you implemented that, a) your developers must not care about XSS anymore, they just use the framework and b) you can sleep better at night, because it is not likely probable, that your site is vulnarable against XSS. Sometimes you want to allow HTML-code passing to the template. That’s ok, just give the developer a chance to avoid mocking or escaping. If you want to audit your code for XSS security problems, just grep for the method signature.

Filed on 22-10-2007, 14:02 under , , , , , & four comments & no trackbacks

XSS-exploits filtered by person 0

xssed.com is a directory for XSS-exploits You can submit your exploits and they got listed and – most important – a copy of the exploit is archived. Of course it is a bit a challenge but it is also useful to keep track with the vendor reactions on your exploits. There is one thing missing, a personalized RSS-feed, which just renders the exploits of a certain author. But there is Yahoo Pipes, at the end a pretty impressive program hosting service, which basically provides tools, filters and operators to be chained to remix content. I’ve played around with it months ago, but never had a real usecase. Now I had and I implemented a pipe for creating an newsfeed from xssed.com. As a result, you can follow my exploits on the startpage.

Update xssed.com is currently blocking requests from the user agent Yahoo Pipes 1.0 which I find pretty sad. I try to get in contact with the xssed.com staff.

Filed on 12-10-2007, 01:01 under , , , & no comments & no trackbacks

On examples 1

One thing I’ve learned is, that one of the hardest things to create are examples. Most people do not use examples as what they are, they use it as templates. So examples have side-effects. You write an example to demonstrate how a certain component works or should be used, but you did not follow the coding style of your organization unit. Developers then going to copy your stuff and bang, your errors are repeated again and again. Or, just do it like Yahoo: add a small piece of PHP for demonstration purposes and introduce a huge security problem. Here it is. Lucky for them, they seem not to use the same code in their live example.

Filed on 27-09-2007, 10:10 under , , , , & one comment & no trackbacks

Planet Websecurity launched 0

Christian Matthies, one of the developers of PHP IDS, launched Planet Websecurity today. Good shit!

Filed on 29-06-2007, 00:12 under , , & no comments & no trackbacks

Easy friend-finding for social retarded 0

Found an XSS-injection on Qype which has enabled a user to inject malicious JavaScript into his profile in order to automatically become a friend of every visitor. As Qype is implemented in Rails they are using Prototype as a JavaScript library which made it pretty easy to implement a fitting exploit:

"&gt;&lt;script&gt;new Ajax.Request('/contact/create?to_user_id=16992', {method: 'post'})&lt;/script&gt;

Qype has fixed the issue.

Filed on 26-06-2007, 10:10 under , , & no comments & no trackbacks

XSS holes on dapper.net 0

Dapper is a web service which provides webservice creation on the fly. You can create your own APIs, feed etc. by just meshing selected areas from different websites. It is pretty similiar to Yahoo Pipes. Switch/Twitch already pointed out, that dapper completely breaks the same origin policy, which is the basic security concept for rich web applications (it is partly broken by Flash anyway, but this is written on another sheet of paper). But even worse, dapper itself was vulnarable against XSS injections which I found out two weeks ago. The vendor replied quickly and fixed the issues I had demonstrated. The combination of breaking the same origin policy and vulnarabilities on dapper is pretty dangerous. Hopefully the developers really know that they are playing with fire.

Filed on 20-06-2007, 20:08 under , , , , & no comments & no trackbacks

Intrusion detection for PHP 4

A local startup, Ormigo, launched PHP IDS, a PHP-based intrusion detection system. It was written by Mario Heiderich and christ1an (who did the regular expression magic). The outstanding difference to approaches like mod_security is that it is purely PHP-based and can be integrated into your application. It basically takes a user submitted content array ($_POST, $_GET, $_COOKIE) and applies a certain set of regular expressions. When I read about it on Oliver Thylmanns Weblog I was really fascinated by the idea and started playing around with it. I’ve submitted a number of patches and proposals and started a branch to introduce a more convenient result handling which was merged back into the trunk a few hours ago. There will be a release in the next weeks and I’m looking forward on helping to make this project grow up.
Just for the record: after christ1an asked me for whom I’m working for he found a nice XSS on Neu.de which was fixed on monday. Thanks again for the hint, christ1an.

The PHP IDS subversion repository: http://phpids.googlecode.com/svn/trunk/

Filed on 15-05-2007, 22:10 under , , , , , & four comments & no trackbacks

XSS-Sunday 5

So, es ist mal wieder XSS-Sonntag und heute beschäftigen wir uns mit einem gänzlich unbekannten Portal: telefonbuch.de. Ein sehr praktischer Service für das Finden von Telefonnummern, wenn man so ein schlechtes Zahlengedächtnis hat wie ich oder schon immer mal im Seitenkontext von telefonbuch.de JavaScript ausführen wollte. Und das geht so (folgenden String in das Formularfeld »Name/Begriff oder Telefonnummer« einfügen):

';alert ("Manfred Krug" );test= '

Die komisch platzierten Leerzeichen werden benötigt, um den die Normalizer-Komponente ein wenig auszutricksen. Interessant auch, dass die Applikation selbst User-Input richtig handhabt, nicht aber das Werbetag, das hier exploitet wird.

Update 12. März 2007:
Hersteller hat Fehler behoben.

Filed on 04-03-2007, 19:07 under , , , & five comments & no trackbacks

Retiring from the retirement? 0

In the last days there was a big discussion in the PHP-community about Stefan Essers retirement from the PHP Security Team. He states, that changing PHPs security from inside is futile, as he has been quoted by Slashdot. Stefan has found a lot of serious security holes in diverse software projects over years and he seems to be one of the best-skilled security researchers who permanently audits free software, which is always a good thing. He critizises, that fixing security holes in PHP takes too long and that he is taken as a persona non grata because he also blames PHPs design problems not just application bugs.
Stefan constantly put the finger on the wound in the last years, that PHP comes with bunch of features making it easy for beginners to blow away their whole leg when writing PHP-programs or being unexperienced in configuring their PHP-environment in a sane way. Starting from remote file inclusion, which is maybe the top-seller of remote exploits in PHP projects, over header-splitting attacks, to register globals he insists that not everything is in the application developers responsibility. Also he did research in PHPs »deep shit«, e.g. memory management. He is one of the authors of hardened PHP, a patch to make a PHP-installation more robust and suhosin, which attempts the same target and being binary compatible to vanilla PHP, which the hardened patch was not. I’m using the suhosin extension and also I really do not agree with the way to low defaults, I like the extension.
I’m not a PHP-insider, I cannot assess the different standpoints but I’m a software developer who likes to use PHP for daily work, for fun and for profit. And I care about application- and interpreter-level security. So my interest is to be sure that the best are working together in the PHP Security Team. Hopefully in a few weeks when heat of the discussion is gone a wise man could bang heads together again. It would be the best for the PHP-world and I would like to ask everyone not to make the gap deeper.

Filed on 20-12-2006, 01:01 under , , , , & no comments & no trackbacks

PHP 5.1.3 released 0

Normally issues like aren’t my topic. Transcripting news from other sites is not minebut the new PHP-version of the 5.1-series provides some important fixes. Deprecation-notice when using the keyword »var« has been removed, so you can use Smarty in combination with PHP5 without the heap of error-messages, fixed buffer-overflow in wordwrap(), XSS-troubles in phpinfo() are fixed and safe-mode reflects itself and checks the first parameter of copy().

Filed on 02-05-2006, 11:11 under , , , , & no comments & no trackbacks

April, April 0

Machte meinen Tag:

From: Georgi Guninski
To: full-disclosure@lists.grok.org.uk
Subject: [Full-disclosure] Linus mass killing integer overflows
the news is, the benevolant dictator has said “let there be C++”, and there is more secure, full featured, reliable and faster linux kernel written2 mainly in C++. the official release is scheduled for 2.8 or when redhat™ becomes ready for the desktop3, whichever comes first.

key improvements include:

a) integer overflows were PITA for the kernel janitors. once the classes SafeInt and SafeLong were implemented with suitable operators, the new kernel is 100% “int/long too big” free. the refactoring tool made this part easy.
b) some clever abuse of exceptions dramatically reduces the amount of OOPS:
cases like ‘(SafeInt)0=foo->bar()’ are now gracefully catch()ed, killing
the OOPS.
c) kernel structures were just lame emulation of C++ objects. now they are native C++ objects.
d) exceptions result in cleaner, easier to read code and almost stop the nasty abuse of “goto”

currently there are discussions for implementing COM in the kernel and/or scripting the kernel from userland, but Linus hasn’t made up his mind yet.

the first public prerelease will be available from ftp://ftp.kernel.org/pub/linux/kernel soon.

My wishlist for 2.8:
-jvm (should boost my swing apps)
-sql support to query kernel table and fs data
-lisp and prolog interpreters (don’t have a use for this but I’m sure somebody will need it)

The kernel folks have taken a seriously wrong turn here. They should have delayed the preview release until the garbage collector was ready. That’s where the real value for these patches starts.


> My wishlist for 2.8:
> -jvm (should boost my swing apps)

agreed. i vote for implementing parrot – http://www.parrotcode.org/ to avoid
perl vs python vs java flames.

> -sql support to query kernel table and fs data

this seems a bad idea. what if a mcse discovers a sql injection in kernelspace? the side effects may be worse than a cake on b1l1.

> -lisp and prolog interpreters (don’t have a use for this but I’m sure
> somebody will need it)

rms likes lisp, so we can count on him.

Filed on 01-04-2006, 20:08 under , , , , , & no comments & no trackbacks

Newer Entries ↘