Fuzzing Perl: A Tale of Two American Fuzzy Lops


tl;dr Over the course of 48 hours, AFLFast found 6 unique flaws in Perl, while AFL 2.32b found 4, all of which AFLFast failed to find.


Abstract

Today I'm going to talk about my experiences fuzzing Perl over the course of 48 hours; 24 of which were spent with the AFLFast fork of AFL 2.30b, and 24 of which were spent with AFL 2.32b, a new version of AFL that has tweaks to the -d "quick & dirty mode" where it skips deterministic steps in order to better emulate the behavior of AFLFast. Michal Zalewski calls this FidgetyAFL.

AFLFast claimed in an email to the AFL-users mailing list:

to have exposed several previously unreported CVEs that could not be exposed by AFL in 24 hours and otherwise exposed vulnerabilities significantly faster than AFL while generating orders of magnitude more unique crashes.

I wanted to test these claims, so what follows is a highly unscientific comparison of the two. Your mileage may vary, but you can read more about AFLFast in their CCS'16 Paper entitled "Coverage-based Greybox Fuzzing as Markov Chain". You can also read up on AFL's technical details and historical notes.


Why Perl?

I chose Perl for a variety of reasons, primarily:

  • A large code base with a wide variety of attack surfaces
  • A friendly and helpful core team of developers

I've been using AFL for several years, and in that time I've helped find, fix and close well over 100 bugs (mostly null pointer derefs, assertion failures, etc), so I know what to expect from AFL and Perl and their dev team.


The First 24

The first 24 hours were spent fuzzing Perl, built from git source, with AFLFast and a starting corpus of the usual 55 or so test cases I handpick for their ability to repeatedly trigger bugs in Perl.

A sample Perl test case:

This example is why I sometimes joke about my cat being able to create perfect Perl code by running across my keyboard. :joy:

$~='`';$_=$:=$~|'%';$;=$^='/'|$~;$;++;$\=$~|"'";$;++;$:.=++$;;$/=++$;;+$\++;$_.=
'#'|$~;$,=++$/;$_.="$\$^$\"";++$,;$_.='@'|'*'&~'!';$_.="$,$;$/$\"";$_.+='!.'|$~.
$~;$_.="$^$/$\$:$\"";$_.='@'|':'&~'*';$_.=$:;$_.=$^&'|';$_.=$".$\;$_.=+"$~$~$~"|
'!#+';++$.;$.++;`$_$:,>&$.`;

I compile Perl the same way every time:

./Configure -des -Dusedevel -DDEBUGGING -Dcc=afl-clang-fast -Doptimize=-O2\ -g && AFL_USE_ASAN=1 make -j2

And I run Perl with AFL the same way every time:

AFL_PRELOAD=/home/geeknik/aflfast/libdislocator/libdislocator.so afl-fuzz -t50+ -m none -i in -o out ./perl @@

afl-plot after 24 hours:

Might be a bug in AFLFast, but when you first start it up, it has a very high number of pending favs, which throws this first graph out of wack.

AFLFast screenshot:

fuzzer stats:

start_time: 1471592218
last_update   : 1471678622
fuzzer_pid: 1014
cycles_done   : 8
execs_done: 14732904
execs_per_sec : 184.69
paths_total   : 24147
paths_favored : 1870
paths_found   : 24092
paths_imported: 0
max_depth : 38
cur_path  : 23211
pending_favs  : 1446
pending_total : 16332
variable_paths: 24142
stability : 86.95%
bitmap_cvg: 58.14%
unique_crashes: 331
unique_hangs  : 364
last_path : 1471678615
last_crash: 1471678242
last_hang : 1471678601
execs_since_crash : 66182
exec_timeout  : 50
afl_banner: perl
afl_version   : 2.30b
command_line  : afl-fuzz -t50+ -m none -i in -o out ./perl @@

Takeaway:

After 24 hours, AFLFast declares we have 331 unique crashes. Upon triage, only 6 of these were unique and serious enough to warrant emails to the Perl security team:

  • 3x heap-use-after-free (all unique locations)
  • 3x heap-buffer-overflow (all unique locations)

This is a pretty good showing by AFLFast, especially with 8 full cycles being completed in just 24 hours. Past Perl fuzzing sessions have dragged on for weeks without completing a single cycle (or generating a single crash). Now lets move on to AFL 2.32b with the FidgetyAFL tweaks.


The Next 24

Perl was recompiled with AFL 2.32b (FidgetyAFL) using the same source as before, no changes were pulled in, no files were edited, etc. The same starting corpus was used and the only other change was the AFL command line:

AFL_PRELOAD=/home/geeknik/aflfast/libdislocator/libdislocator.so afl-fuzz -d -t50+ -m none -i in -o out ./perl @@

afl-plot after 24 hours:
afl-plot after 24 hours

FidgetyAFL screenshot:
FidgetyAFL screenshot

fuzzer stats:

start_time: 1471680593
last_update   : 1471766996
fuzzer_pid: 14641
cycles_done   : 0
execs_done: 16938116
execs_per_sec : 60.64
paths_total   : 24461
paths_favored : 2408
paths_found   : 24406
paths_imported: 0
max_depth : 13
cur_path  : 23749
pending_favs  : 245
pending_total : 19469
variable_paths: 24456
stability : 84.09%
bitmap_cvg: 59.79%
unique_crashes: 662
unique_hangs  : 500
last_path : 1471766988
last_crash: 1471763507
last_hang : 1471733050
execs_since_crash : 676444
exec_timeout  : 50
afl_banner: perl
afl_version   : 2.32b
command_line  : afl-fuzz -d -t50+ -m none -i in -o afl232-out ./perl @@

Takeaway:
After 24 hours, FidgetyAFL declares we have 662 unique crashes. Curiously we have exactly 2x more crashes than AFLFast. Upon triage, only 4 of these were unique:

I thought after finding the null ptr deref, we were on a new track and would start finding some more interesting things. I started noticing more and more of these Bus error so I decide to look closer. Run it under gdb and I see SIGBUS. I had to do some quick reading, ask a few questions. I'm pretty sure I dismissed those earlier as just extraneous DEBUG output, so I've made a note to go back and check for others.

While not able to complete a full cycle during the 24 hour period, this is not a bad showing by AFL 2.32b compared to prior fuzzing sessions without the FidgetyAFL tweaks, which could go on for days or weeks before finding a single crash.


The Verdict

As you can see from this completely unscientific study, AFLFast is, in my opinion, neither better or worse at finding a wider variety of security flaws when compared to AFL 2.32b over the same period of time. And while AFL 2.32b managed to find more unique crashes overall with 662 compared to AFLFast's 331, I feel like they all kept pointing to the same line of code which means they aren't all that unique.

Perl has many places for bugs to hide, some have been lurking for many, many years. I'm going to keep fuzzing with a mixture of both AFLFast and AFL 2.32+ because 11 (see note) potential vulnerabilities in 48 hours is a mighty good number of holes to patch. A Perl developer I spoke with said they are interested if perl -D crashes, so that's a good place to start if you want to join in on the fun.

Note: During the same span of time (the 48 hours prior to this being written), AFLFast triggered a null ptr deref + segfault in PHP 5.6.25 using a random corpus from 3v4l.org (which in my opinion is a gold mine for anyone looking to fuzz PHP).


Errata

I did not count the many Assertion Failures which occurred during these 2 rounds of fuzzing, mainly because they have no impact on security, and since this is a -DDEBUGGING build of Perl, asserts can frequent fail. I may go back and submit them to the dev team at a later point if I have time.

I am positive that I missed some SIGBUS faults in both test sessions, so I will be going back over the fuzzing output again in the coming days, and submitting any new flaws that I find.

All flaws found in this test session have been reported to the Perl security team and/or the public Perl bug tracker.


Technical Details

All fuzzing was conducted on $5/mo DigitalOcean Debian 8.5 x64 VMs with the bare minimum software necessary to compile and fuzz Perl.


Credits

Without these people, this wouldn't have been possible:


License

MIT License - Copyright (c) 2016 Brian 'geeknik' Carpenter

Last updated: 8/21/2016 03:23:54

Geeknik Labs

Also on this blog

SHARE:  Email · Facebook · Google · Twitter · Tumblr · Kindle
SUBSCRIBE:  Receive an email on new posts from Geeknik Labs

Comments

Interesting Experiment. And helpful results.
2016-08-21, Anonymous

It's frustrating to know. The errors.
2016-08-21, onlibuton

Helpful write up!
I'm curious: how did you triage the crashes and determine how many truly unique failures were found?
2016-08-22, Brad

ASAN output is predictable and you can make basic triage scripts that look for a unique signature or a unique pattern. By doing that and a little manual verification, I was able to accurately count how many truly unique failures happened over the course of those 24 hours.
2016-08-22, Geeknik Labs

How did you determine which crashes were use after frees? That sounds like something you'd have to manually verify.
2016-08-23, John

ASAN (Address Sanitizer) will tell you all about it when you run a build compiled with support for it. Here is sample output from another bug:
==7138== ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1337e70060 at pc 0x7f135a3fc2dd bp 0x7fff714b1cf0 sp 0x7fff714b1ce8
READ of size 8 at 0x7f1337e70060 thread T0
2016-08-23, Geeknik Labs

This is interesting. For fuzzing the programming languages, are you using any AST?
2016-08-25, AJ

Very cool results:) I have tried a few things:
http://pastebin.com/BTwCyHet
2016-08-25, Trap 0x86

Nice! I hope you submit them and they aren't duplicates. ;)
2016-08-25, Geeknik Labs

Also no on a nit nuk nut.
2016-10-06, onlibuton


  • Notify me upon new comments

☺ Got it