So what is a URL preview?

A URL preview is something which we’ve all seen while we browse the internet. Somebody shares a URL to some piece of content, usually a web page, and the social platform which we’re browsing automatically generates a preview of the content found at the URL. In the example Facebook post below, I am sharing a URL to one of my favourite games, RuneScape:

These previews are generally very helpful and provide us with rich context surrounding what sort of content we can expect to find if we were to actually click on the URL. In the example above, we get several pieces of information:

• A main picture which is supposed to depict what we might see when we click the link
• The actual domain name which we can expect to find ourselves at when we click the link
• A short textual description of what the web page is about

Upon seeing the above post you might expect that clicking on the post will take you to runescape.com and that the web page will look roughly as you expect it to look based on the preview provided by Facebook. However if you go ahead and actually click on the post above, you’ll find yourself at my proof-of-concept website instead: poc.razb.me

How are URL previews created?

In order to understand how a URL preview might be spoofed, we must first understand how a URL preview is created. I will be using Facebook as my example platform, but all platforms which provide this sort of preview will function in a similar manner.

There are really only three main steps in generating a URL preview:

1. Download a copy of the web page found at the URL
2. Inspect it for various metadata tags
3. Generate the preview

The first step should be pretty straightforward, just make a GET request to the URL and save the resulting HTML.

The second step is specific to the platform. Each platform defines its own set of metadata tags which can be added to the HTML of any webpage for providing information to be used for populating URL previews. You can find out all about these HTML tags here: https://developers.facebook.com/docs/sharing/webmasters/

The third step is to put together the information provided via the metadata tags into a visual format which becomes the URL preview.

How can we spoof the preview generation?

So far everything looks fine. I share a URL, Facebook fetches that web page and parses the HTML tags then generates the preview from them. How can we possibly spoof any of this? We don’t necessarily own the web page which we’re linking to, so what can we do?

Let’s try sharing a URL to a web page which we do own and check the GET request which Facebook makes.

Upon inspection, when Facebook makes a GET request to my proof-of-concept domain, the headers look something like this:

	{
	“Host“: “poc.razb.me“,
	“Connection“: “Keep-Alive“,
	“Accept-Encoding“: “gzip“,
	“Accept“: “*\\/*“,
	“User-Agent“: “facebookexternalhit\\/1.1 (+http:\\/\\/www.facebook.com\\/externalhit_uatext.php)“,
	}

view raw facebook_header.json hosted with by GitHub

The field worth noting is the “User-Agent” field. This is used as a courtesy by the client requesting our web page and it lets us know what sort of browser (or in this case bot) was used to access our web page.

In the case of the Facebook URL preview generation, Facebook’s bot configured the “User-Agent” field to something specific to their use case and they call it “facebookexternalhit”.

For a normal user like you or I requesting my web page, the header will look something like this:

	{
	“Host“: “poc.razb.me“,
	“Connection“: “Keep-Alive“,
	“Accept-Encoding“: “gzip“,
	“User-Agent“: “Mozilla\\/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit\\/537.36 (KHTML, like Gecko) Chrome\\/71.0.3578.98 Safari\\/537.36“,
	}

view raw normal_header.json hosted with by GitHub

In the normal case, the “User-Agent” field looks very different and actually contains information about the browser which I used to access my web page.

It turns out that we can use this subtle difference in User-Agent to serve Facebook’s bot a different response than we serve normal users.

User-Agent switching

If for every request we read the User-Agent field before serving our web page, we can switch on the User-Agent value to serve different responses for bots and for regular users.

So what might we do differently for a bot so that it generates a preview for something other than our current web page? A simple solution is to simply return a 300 redirect to your target web page of choice.

Here is some sample code which can do just that in PHP:

	<?php
	$user_agent = $_SERVER[“HTTP_USER_AGENT“];
	if (strpos($user_agent, “facebookexternalhit“) !== false) {
	header(“Location: https://runescape.com“);
	} else {
	echo(“poc.razb.me“);
	}
	?>

view raw fb_bot_redirect.php hosted with by GitHub

In this example we simply set the “Location” header response to https://runescape.com. This will tell Facebook’s bot to ignore whatever content it found at my URL poc.razb.me, and instead go looking at runescape.com. We can confirm that the bot is doing what we expect by using Facebook’s sharing debugger: https://developers.facebook.com/tools/debug/sharing/?q=https%3A%2F%2Fpoc.razb.me

It turns out this is all it takes to spoof the Facebook URL preview bot. It is also possible to perform the same simple switch on iMessage, Twitter, and Reddit previews. Here are some examples:

It’s important to note that both iMessage and Reddit actually preserve the original URL when generating the preview. This is a very good practise as it makes this sort of attack much less effective when the user can still see the original URL.

However, in the case of Facebook and Twitter they use the redirected URL in the preview and omit the original URL entirely. This makes it possible to trick somebody into thinking that they are about to go to one domain when in fact they will be taken to another.

Responsible Disclosure

Prior to posting this blog post, I reached out to Facebook and Twitter (via their respective bug bounty programs here and here) and made sure that they are aware of the trick that I outlined in this post. They both got back to me and said that it is too low risk and they have no plans on changing the current behaviour.

So for the time being, pay extra close attention when clicking on content on Facebook and Twitter, and make sure the domain you end up at is the one you expected.

Since its initial release on July 10, 2008, the iOS App Store has amassed over 2.2 million apps. Combine that with a study by Nielsen in 2015 which shows that the average smartphone user interacts with 10 apps a day, and 30 apps a month for a total of over 35 hours spent using apps every month, it’s safe to say that we love our apps.

The way we interact with all of these apps is generally the same: download it from the App Store, find the app’s icon on your home screen, tap it and it magically opens. On the surface, there doesn’t seem much to an app. They all just sit there patiently on your home screen, icons waiting for you to tap on them and bring them to life. But have you ever wondered what actually goes on under the hood? How these apps work, and what they’re made of? Let’s take a look.

Note: throughout this post we will be looking at specific examples from an app which I wrote: RS Xchange.

The IPA file

After finding a new app which you’d like to try out on the App Store, you press the download button and wait a little while for stuff to happen. During this time, iOS is downloading the app in the form of a file with an extension of “.ipa”. So what exactly is this “.ipa” file?

“.ipa” files are actually just “.zip” files. Renaming .ipa to .zip will allow you to extract the contents of the .ipa file in much the same way you extract .zip files. After extraction, you will find a folder with the following structure:

• iTunesArtwork
• iTunesMetadata.plist
• Payload/
• META-INF/

Let’s run through this from top to bottom.

iTunesArtwork

This file is actually a .png file. Adding the “.png” extension to it will allow you to open the file with your image viewing tool of choice. Upon opening the file, you will see that this is just the app’s icon as it is shown on the App Store. This image will have a size of 1024×1024. Here is an example of an iTunesArtwork file:

iTunesArtwork for the RS Xchange app

iTunesMetadata.plist

This file contains lots of information about the app, its developer, when it was downloaded, and which account it was downloaded by. This is what a sample iTunesMetadata.plist file might look like:

Payload/

The Payload/ folder contains the actual app binary along with all of its resources that it needs to function. Once inside the Payoad/ folder, there will be a single file with the “.app” extension. This file is actually just another folder, and you can open it like any other folder. On MacOS, right click it and press “Show Package Contents”.

The contents of this folder will differ from app to app. This is the stuff that defines each individual app. Here is an example of some of the things you might find:

• AppIcon29x29.png
• [email protected]
• AppIcon29x29@2x~ipad.png
• [email protected]
• AppIcon29x29~ipad.png
• [email protected]
• AppIcon40x40@2x~ipad.png
• [email protected]
• AppIcon40x40~ipad.png
• AppIcon50x50@2x~ipad.png
• AppIcon50x50~ipad.png
• AppIcon57x57.png
• [email protected]
• [email protected]
• [email protected]
• AppIcon72x72@2x~ipad.png
• AppIcon72x72~ipad.png
• AppIcon76x76@2x~ipad.png
• AppIcon76x76~ipad.png
• Assets.car
• Base.lproj/
• Frameworks/
• Info.plist
• ItemDetailsStoryboard.storyboardc/
• LaunchScreen.storyboardc/
• OSRSItemDetailsDescriptionTableViewCell.nib
• OSRSItemDetailsHeaderView.nib
• OSRSItemDetailsWikiaTableViewCell.nib
• OSRSItemSearchTableViewCell.nib
• PkgInfo
• RS Xchange
• SC_Info/
• TSMessagesDefaultDesign.json
• _CodeSignature/
• archived-expanded-entitlements.xcent

Let’s run through them top to bottom.

AppIcon.png

First, you will notice a bunch of images named something like “AppIcon<something>.png”. These images correspond to, as you might have guessed, the app’s icon as you see it on your home screen. There are lots of various sizes of this icon, each one optimized for a different device and different place that it might be shown. For example, iPads have different app icon dimension requirements than iPhones, and the app icon as it is shown in a notification has a different dimension requirement than the app icon which is shown on the home screen.

Assets.car

This file contains some (or all) of the images which the app might use to display once you open it. The images are stored in an optimized format, and you need special tools to take them out and look at them one by one. One such tool which can do this is called the Asset Catalog Tinkerer. Upon exporting, you will get a list of image files which you can open with your image viewing tool of choice. Here is an example of some of the images which you might find inside of the Assets.car file:

[See image gallery at blog.razb.me]

Base.lproj/

This folder has to do with localization. If an app supports multiple languages, each resource which needs to be translated (strings, images, storyboards, xibs) will go into its own folder which ends in “.lproj”. For example, if your app was localized to english and french, you will have three “.lproj” folders: en.lproj, fr.lproj, and Base.lproj. The en.lproj and fr.lproj folders will contain resources which have been converted to their respective languages, and Base.lproj contains the default resources to use in the case when a user’s phone is using a language which is not supported by the app. In most cases, Base.lproj just contains the english version of the app’s resources.

Frameworks/

This folder contains all of the dynamic frameworks and libraries which the app relies on. This is just some code which was written outside of the main app, and which is used by the main app. Here is an example of some of the files which might be found in this folder:

• AFNetworking.framework/
• Blindside.framework/
• BlindsidedStoryboard.framework/
• ChameleonFramework.framework/
• Charts.framework/
• DZNEmptyDataSet.framework/
• FontAwesome_iOS.framework/
• Harpy.framework/
• HexColors.framework/
• KSDeferred.framework/
• MaryPopin.framework/
• NXTSegmentedControl.framework/
• PPiAwesomeButton.framework/
• PPiFlatSegmentedControl.framework/
• SCLAlertView_Objective_C.framework/
• SDWebImage.framework/
• TSMessages.framework/
• iRate.framework/
• libswiftContacts.dylib
• libswiftCore.dylib
• libswiftCoreGraphics.dylib
• libswiftCoreImage.dylib
• libswiftDarwin.dylib
• libswiftDispatch.dylib
• libswiftFoundation.dylib
• libswiftObjectiveC.dylib
• libswiftUIKit.dylib

Amongst these, you’ll notice that some apps contain files which start with “libswift”, and have the extension “.dylib”. These files make up the Swift runtime, and they are necessary to be included in every app which contains Swift code.

Info.plist

This file contains generic information about the app itself. Things ranging from the app’s name, to the AppIcon files we talked about above, to the permissions which the app might ask for (photos, camera, contacts, etc), and even some internal information such as the unique identifier for the app (called the bundle identifier). Here is a sample Info.plist file:

A lot of interesting information can be found in the Info.plist file. An official summary of everything this file might contain can be found here and here.

<something>.storyboardc/

Folders which end in .storyboardc contain some information related to the layout of particular screens or views within the app. The presence of these folders indicate that the app was developed using the Interface Builder tool. Within these folders, you will find several .nib files along with another Info.plist file:

• Info.plist
• UIViewController-x1L-Z2-jkR.nib
• x1L-Z2-jkR-view-7LY-9B-3EQ.nib

The Info.plist file included here is in binary format, and you need to convert it to XML format before being able to read it like normal .plist files. Run this command to convert it to XML:

plutil -convert xml1 Info.plist

Here is what an example Info.plist file could look like inside of a .storyboardc file:

The keys/values within these Info.plist files describe some information relating to the .nib files found alongside them. In this case, the UIStoryboardDesignatedEntryPointIdentifier key is indicating that the view controller with the identifier UIViewController-x1L-Z2-jkR is to be used as the “entry point” for the application, or the first screen to be displayed when the app is opened. The UIViewControllerIdentifiersToNibNames key is mapping identifiers in the compiled .nib files to other .nib files.

<something>.nib

.nib files are the result of compiling .xib files using the ibtool utility. .xib files are XML files which define the user interface for a specific element within the app. Everything from individual buttons to entire screens can be defined in .xib files. Here is a sample .xib file:

And the corresponding user interface it produces:

.xib user interface

It’s important to note that while developing, the XML of a .xib file is never modified directly by a person. The resulting XML is generated through the Interface Builder tool.

PkgInfo

This is an auto-generated file which contains the 4-byte package type followed by the 4-byte signature of your application. For iOS apps, the package type will be “APPL“, and the signature will be 0x3f3f3f3f.
More information about this file can be found here.

RS Xchange

This file name will be different from app to app, and the name corresponds to the value for the CFBundleExecutable in the Info.plist file. This is the app’s main executable. It contains all of the code necessary to make the app run. This is the file that gets executed when you tap on the app’s icon on your home screen.

When you download an .ipa file from the App Store, this executable is encrypted, so you can’t take a closer look at what’s inside of it. Fortunately there is a way to get the decrypted version, but we’ll cover that in a future blog post.

SC_Info/

This folder contains keys which are used for decrypting the app executables. The contents will look like this:

• Manifest.plist
• RS Xchange.sinf
• RS Xchange.supf
• RS Xchange.supp
• RS Xchange.supx

The .sinf, .supf, .supp, .supx files are used for decrypting, and the Manifest.plist file is used to list all of the relevant .sinf files which are to be used. Here is a sample Manifest.plist file:

TSMessagesDefaultDesign.json

This file is specific to the RS Xchange app which we have been inspecting throughout this post. It is just one of the additional resources which are included with the app. Apps can have all sorts of arbitrary resources which they require to function, and you will find them all in the root of the .app folder.

_CodeSignature/

This folder contains information related to code signing. When a developer decides to distribute an app via the App Store, code signing is performed on the executable and all of the other resources included within the app. The purpose of this process is to ensure that the app can not be modified by any malicious third party after the developer has chosen to distribute a version of their app.

There is a single file within this folder called CodeResources. Taking a closer look at this file, it is actually a .plist, and can be opened the same way as any other .plist file. You can find a sample CodeResources file here. (I would include it directly here, but it’s 3000+ lines long)

Each file which is eligible for code signing is included in this file, along with the corresponding hash data which should result from the process of signing each file.

If any of these files are modified after the developer has signed it, the hashes of the files will differ and the app will be unable to be run on a device.

More information about code signing can be found here and here.

archived-expanded-entitlements.xcent

This file is once again just a .plist file in disguise. It contains all of the entitlements which the app has requested, such as push notifications and access to iCloud storage. Here is a sample archived-expanded-entitlements.xcent file:

In this example, the only entitlement the app is requesting is for keychain-access-groups. You can find a complete list of the entitlements which an app may request here.

META-INF/

This folder contains some general metadata about the .ipa file. Under the META-INF/ folder, you will find two more files:

• com.apple.FixedZipMetadata.bin
• com.apple.ZipMetadata.plist

You typically don’t need to worry about this folder, and there isn’t much documentation to go with it, but let’s run through the two files found here anyway.

com.apple.ZipMetadata.plist

The com.apple.ZipMetadata.plist file contains some information about how the .ipa file was created, along with its uncompressed size. This is what a sample com.apple.ZipMetadata.plist file might look like:

Note that this .plist file is typically found in binary format, and you will need to do some work to convert it to XML format to get the output above. Converting a binary plist to an XML plist can be done with the following command:

plutil -convert xml1 com.apple.ZipMetadata.plist

com.apple.FixedZipMetadata.bin

The next file, com.apple.FixedZipMetadata.bin, is binary file of some sort. There isn’t much documentation about what this is or how to read it, but doing a hexdump on it yields the following:

00000000  4d 64 46 78 01 10 00 00  00 00 00 00 00 00 00 00  |MdFx............|
00000010  00 00 00 00 00 00 00                              |.......|
00000017

That’s all folks

We’ve officially gone through everything that is found within a typical App Store app file which you download and install on to your device. All of these components work together to provide you with all of the functionality you know and love, like lining up candy, browsing memes, and of course, colouring pictures of cats.

The Swift Life” hit the app store.

The app is geared towards social media, allowing you to post content and interact with other people’s content in much the same way as Facebook, Instagram, and all the typical social media platforms we’re all familiar with. However, the one thing that seems to set her app apart is the fact that when a post by a fan gets popular enough, there’s a chance that Taylor Swift herself will see it, and even possibly reply. Cool stuff.

Aside from that, the app is jam-packed with Taylor Swift-ness; from exclusive pictures and videos posted by the Swift goddess herself, to animated Taylor-themed emoji (which they’ve creatively dubbed Taymoji), to having Taylor’s songs with lyrics constantly playing at the top of the app. Even the error messages make use of lyrics from Taylor’s songs.

“Blank Space” error message

Now the question is, with all of this Swiftiness packed into one app, surely it must also be written in Swift, right? Wrong.

Taylor Swift’s “The Swift Life” is, ironically, not written in Swift.

Some technical details:
There are two main programming languages which are used to write iOS applications: Objective-C and Swift.
In order for an iOS app to support Swift code, it must include something called the Swift runtime which is just a collection of frameworks conveniently packaged under a folder called “Frameworks” inside of the iOS App Store Package (.IPA) file. For a more detailed look at the inner-workings of an .IPA file, check out this post. Taking a look at Taylor’s app, it does not contain the Swift runtime, and therefore it cannot contain any Swift code.

Quite a missed opportunity if you ask me.

I guess the developers just weren’t ready for it.

According to this post on the Apple Developer forums regarding the issue discussed in this topic,

This is a bug in the beta that should already be fixed by the time you read this.

kSecAttrAccessGroupToken is writable only by CryptoTokenKit smart card drivers. Apps can query the keychain using that attribute in order to find items stored on a particular smart card. This attribute is not for any other use.

It appears as though the kSecAttrAccessGroupToken access group was never meant to be writable by any app outside of CryptoTokenKit. This means that normal apps like yours and mine cannot make use of this access group for the purpose of writing our own custom data to it.

It’s also worth noting that the keychain autodelete behaviour which was observed in the iOS 10.3 beta did not end up making it into the public releases. So, for the time being, our beloved keychain continues to persist data beyond app deletion.

Original post below:

On February 7th, 2017, Apple released iOS 10.3 Beta2 to the public, and with that came some changes to the keychain.

The keychain was introduced to iOS devices with the release of iOS 7. The intention of the keychain is to allow a user to securely store login credentials to any number of services so as to not have to type them in every time. Safari uses the iOS keychain and prompts users to save their credentials whenever they login to any websites. However, the keychain is also available to developers. This means that it is possible to store passwords, certificates, and keys without any user interaction.

One of the (potentially unintended) side-effects of the keychain is that anything stored inside of it would persist even after uninstalling the app. This means that anything an app stores in the keychain would potentially live beyond the app itself, and the next time the app is installed and launched, it could read from the keychain and continue as if nothing ever happened.

This property of the keychain is quite useful and heavily relied upon by developers, however, it seems as though this might change with iOS 10.3. As outlined in this post on the Apple developer forums, keychain items are now automatically deleted when the app is deleted. For the purpose of keychain access groups, the keychain items are deleted when the last app in that access group is deleted.

This can be problematic for many apps which rely on the persistence of the keychain in order for their features to work. This was certainly true for me, so I decided to go looking for a workaround.

That is when I stumbled upon the constant kSecAttrAccessGroupToken, of which there does not seem to be much information about on the Internet. Here is the documentation found with this constant:

/*!
     @enum kSecAttrAccessGroup Value Constants
     @constant kSecAttrAccessGroupToken Represents well-known access group
         which contains items provided by external token (typically smart card).
         This may be used as a value for kSecAttrAccessGroup attribute. Every
         application has access to this access group so it is not needed to
         explicitly list it in keychain-access-groups entitlement, but application
         must explicitly state this access group in keychain queries in order to
         be able to access items from external tokens.
*/
extern const CFStringRef kSecAttrAccessGroupToken
    __OSX_AVAILABLE_STARTING(__MAC_10_12, __IPHONE_10_0);

So, it would seem as though there is an access group, introduced with iOS 10, which all apps on the phone have read/write access to. If this is the case, then anything stored within that access group should persist forever, since any app currently installed may want to read from it in the future.

After testing my hypothesis, this indeed seems to be the case. Saving keychain items with the kSecAttrAccessGroupToken access group persists beyond app deletion. I wrote some sample code demonstrating this here.

So, in order to preserve existing keychain functionality and have entries persist beyond app deletion, a simple work around could be to use this access group. However, there is a caveat:

Since any app on the system can freely read and write to this access group, it is important to consider any security side effects this might have for your app. If you are storing sensitive information in the keychain, it could be worth encrypting keychain entries yourself before saving it off to the keychain.

How it works:

You —> Charles —> Internet

By sitting in between you and the Internet, Charles is able to pick up on every single network request that you send and receive. Once it picks up on some http requests, it shows you everything you need to know about it such as the headers in the request and the body of the response. It looks something like this:

This works great for http or unencrypted content, but what about https? Luckily there’s ways around that as well. Without going into the details, by installing a certificate on the device you want to debug, Charles can provide the same functionality for encrypted connections as well. Let’s use the Shopkick app as an example.

Trying to analyze the calls to https://app.shopkick.com doesn’t show any useful information because they’re encrypted over SSL. Let’s enable SSL Proxying.

And now we can see all that traffic in the clear.

Charles also allows us to copy the cURL command for a particular request. Let’s use this to simulate a login from the command line.

razvanbangu@Razvans-MacBook-Pro Fraudkick (master) $ curl -H "Host: app.shopkick.com" -H "X-Sts: 1459834233364" -H "X-Screen: 320,480" -H "X-App: shopkick/4.8.5" -H "X-Device: Apple/Unknown: iPhone4,1" -H "X-Device-Id: enhjbisq9flt3g37hegoctwf1s7gvkhtxks2lzry" -H "X-Sys: iPhone OS/9.0" -H "Proxy-Connection: keep-alive" -H "X-Client-Capabilities: 2" -H "Accept: */*" -H "Accept-Language: en-ca" -H "User-Agent: shopkick/1177 CFNetwork/758.0.2 Darwin/15.0.0" -H "X-Adv-Id: CD5B8AAC-32D4-4988-B217-349F3EDB6887" -H "X-Mac: 3DjMFoLY3Alb1HT-_aizd7FTyBU=" -H "X-API-Key: 0e8450ff-351e-406d-a365-03f735f87d9f" -H "Cookie: app_version=4.8.5; device=Apple/Unknown%3A%20iPhone4%2C1; device_id=enhjbisq9flt3g37hegoctwf1s7gvkhtxks2lzry; device_os=iPhone%20OS/9.0; device_type=Apple/Unknown%3A%20iPhone4%2C1" --data "password=<notmypassword>&device_id=enhjbisq9flt3g37hegoctwf1s7gvkhtxks2lzry&email=raz10%40razb.me&device_model=Unknown%3A+iPhone4%2C1&kcid=6Xl4mn7rEpgHUfmrIwdE203H3iKarm6Gf4%2FVaGH2LyU%3D-6Xl4mn7rEpgHUfmrIwdE203H3iKarm6Gf4%2FVaGH2LyU%3D-6Xl4mn7rEpgHUfmrIwdE203H3iKarm6Gf4%2FVaGH2LyU%3D" --compressed https://app.shopkick.com/shopkick/v1/user/login
{"status": 0, "user_id": 16836985655, "user_info": {"first_name": "razzle", "is_google_plus_connected": false, "is_facebook_connected": false, "country": 1, "created_ts_utc": 1459823387000, "is_registered": true, "enrolled_loyalty_program_ids": [], "email": "[email protected]", "is_buy_and_collect_enrolled": false}, "client_app_ui_flags": {"aggregate_deals_enabled": true, "tab_navigation_enabled": true, "preselect_all_recommended_friends": false, "allowed_invite_media": "1,2,3", "show_kicks_center": true, "use_combined_store_and_deals_tab": false, "max_number_invalid_email_logs": 3, "saving_deals_enabled": true, "default_tab_context_override": "store", "show_new_card_linking_screen": false, "tab_context_order_override": ["store", "deals", "browse"], "number_of_recommendations": 10}, "session_key_base64_enc": "JRtCoHkdBjx6lA7xU+oUa3mgDmoltqNaj6YYvNPSNgbmYR6T49gzHEx6WWIkXUzoRcKgCP/FMmmmvLxtFZYfRg==", "common": {"popups_response": {"force_display": true, "popups": [{"type": 6, "layered_popup_details": {"text_views": [{"text": "Thanks for all your shopkick love - you've logged in on an additional device. Just remember that you can only do walk-ins and scans with one device at a time and can't share devices with your friends.", "frame": {"y": 155, "x": 20, "height": 200, "width": 280}, "style": {"color": "FFFFFF", "font_size": 14, "font_name": "HelveticaNeue"}, "android_style": {"color": "FFFFFF", "font_size": 14, "font_face": "Roboto"}}, {"text": "Did you know?", "frame": {"y": 80, "x": 0, "height": 100, "width": 320}, "style": {"color": "FFFFFF", "font_size": 40, "font_name": "HelveticaNeue-Light"}, "android_style": {"color": "FFFFFF", "font_size": 40, "font_face": "Roboto"}}], "buttons": [{"action": {"type": 1}, "image_view_info": {"frame": {"y": 0, "x": 0, "height": 460, "width": 320}, "image_url": "https://app2.shopkick.com/shopkick/images/generic_buttons/transparent_dismiss.png#2x&1_5x"}}], "image_views": [{"frame": {"y": 0, "x": 0, "height": 50, "width": 50}, "image_url": "https://app2.shopkick.com/shopkick/images/generic_buttons/close_button.png#2x&1_5x"}]}}]}, "request_id": "ChPAjAAAz1wABS+2TB5dSA"}}

Charles is great for debugging mobile apps which mostly only make use of http connections, however in order for this to work on a mobile device we need to configure the device to use our computer as a proxy. On iOS you can do this by going to the wifi settings and scrolling down to the HTTP Proxy section. Fill in the details to match your computer where Charles is running.

Once that is done, open safari and navigate to http://charlesproxy.com/getssl That will prompt you to install the Charles certificate and allow you to debug SSL connections coming from your device.

Happy debugging!

The result is YES under normal circumstances, but we want to test what happens if the result was ever NO.

One way to do that is by dragging the little green arrow next to the breakpoint marker down into the else statement so that it looks like this:

The result is YES, but we have managed to get ourselves into the else condition to test what happens going down that code path.

Upon dragging the green arrow, Xcode will display the following error:

Take a second to read the error and understand it. Then, press ‘Move’ to move the instruction pointer. Now, simply resume execution of the program. Voila, you have just tested your alternative code path without stopping, commenting out or modifying the code, and restarting the app.

In order to achieve this, what we are going to do is link together a chain of computers starting with yours, going all the way around the world, and ending right back where it started; with yours. After setting that up, we will be sending a picture of yourself from yourself to yourself, and in the process circumnavigating the world.

Before we begin, I will be using DigitalOcean to spin up our servers. If you want to follow along at home, DigitalOcean offers new users a promo code to get you up and running for free. Use: DONEWS or you can use my personal promo url: m.do.co/c/74fd80a9022a the choice is yours.

After creating an account with DigitalOcean, you will want to set up an SSH key for use with your servers. DigitalOcean has a great tutorial on how to set that up which you should follow and complete before reading further.

Now, we will spin up 6 servers, one in each of the following data centres: San Fransisco, New York, London, Amsterdam, Frankfurt, and Singapore. Upon creating each server, select Ubuntu 14.04.4, the $5/month option, and add your SSH key.

NOTE: All commands in this tutorial are assumed to be executed by the root user.

Once all of your servers are up and running, SSH into every one of them and set a root password. For the sake of simplicity, we will be communicating between our servers via SSH root@host. You probably do not want to do this in practise.

Before we go any further, it may be worth confirming that a request from Singapore->San Fransisco does indeed go across the Pacific Ocean to complete the loop around the world. In order to do that, SSH into your Singapore machine and type in the following command:

$ traceroute <San Fransisco IP Address> -T

Replacing <San Fransisco IP Address> with the actual address of your San Fransisco server.

If all goes well, you should see output similar to this:

root@AroundTheWorldSingapore:~# traceroute 192.241.221.191 -T
traceroute to 192.241.221.191 (192.241.221.191), 30 hops max, 60 byte packets
 1  128.199.191.253 (128.199.191.253)  0.477 ms 128.199.191.254 (128.199.191.254)  0.474 ms  0.452 ms
 2  103.253.144.233 (103.253.144.233)  0.440 ms 103.253.144.229 (103.253.144.229)  0.449 ms  0.429 ms
 3  116.51.27.189 (116.51.27.189)  1.371 ms 103.253.144.249 (103.253.144.249)  0.440 ms 116.51.27.189 (116.51.27.189)  1.596 ms
 4  ae-1.r20.sngpsi05.sg.bb.gin.ntt.net (129.250.3.146)  14.718 ms 116.51.27.189 (116.51.27.189)  1.692 ms ae-1.r20.sngpsi05.sg.bb.gin.ntt.net (129.250.3.146)  14.708 ms
 5  ae-8.r22.snjsca04.us.bb.gin.ntt.net (129.250.3.48)  167.050 ms  178.774 ms ae-1.r20.sngpsi05.sg.bb.gin.ntt.net (129.250.3.146)  14.732 ms
 6  ae-19.r01.snjsca04.us.bb.gin.ntt.net (129.250.3.27)  181.920 ms ae-8.r22.snjsca04.us.bb.gin.ntt.net (129.250.3.48)  168.298 ms  169.413 ms
 7  ae-40.r02.snjsca04.us.bb.gin.ntt.net (129.250.3.121)  175.240 ms ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  197.201 ms ae-40.r02.snjsca04.us.bb.gin.ntt.net (129.250.3.121)  187.610 ms
 8  xe-0-4-0-17.r06.plalca01.us.ce.gin.ntt.net (129.250.203.82)  168.083 ms ae-4.r05.plalca01.us.bb.gin.ntt.net (129.250.5.32)  182.682 ms xe-0-0-0-23.r05.plalca01.us.ce.gin.ntt.net (129.250.204.118)  181.313 ms
 9  xe-0-0-0-23.r05.plalca01.us.ce.gin.ntt.net (129.250.204.118)  186.122 ms 198.199.99.238 (198.199.99.238)  184.223 ms xe-0-4-0-17.r06.plalca01.us.ce.gin.ntt.net (129.250.203.82)  178.457 ms
10  192.241.221.191 (192.241.221.191)  186.219 ms  176.327 ms  172.506 ms

Taking a look at lines 4 and 5, we can see that our packet went from sg.bb.gin.ntt.net to us.bb.gin.ntt.net and the time across those two nodes spiked from 14.718ms to 167.050ms. This is pretty promising that our packets do in fact cross the ocean. Now we can continue with linking our servers together.

In order to move data from one server to another, we’re going to use SSH port forwarding. In this tutorial I will explain how to do this for the first and last server, and the process for the ones in the middle is pretty much identical.

SSH into all of your servers, modify /etc/ssh/sshd_config, and add the following line to the file:

GatewayPorts clientspecified

After saving, restart ssh:

$ service ssh restart

This allows us to specify which interface the next machine in the chain should listen on.

Now, from your San Fransisco server, execute the following command:

$ screen ssh -L <San Fransisco IP Address>:4455:<New York IP Address>:4455 <New York IP Address>

This tells our San Fransisco server to forward all requests to its port 4455 over to our New York Server’s port 4455. Repeat this process for New York -> London, London -> Amsterdam, Amsterdam -> Frankfurt, and lastly Frankfurt -> Singapore.

If you are using OSX, before going onto the last step you need to allow remote login to your mac. To do this, go to Settings -> Sharing-> Remote Login. Make sure it is checked.

For the last link in the chain, we must hook up our Singapore server to our local computer. From your personal computer, execute the following command:

$ screen ssh -R <Singapore IP Address>:4455:localhost:22 <Singapore IP Address>

At this point, you now have a complete chain of computers starting from San Fransisco, around the world to Singapore, and finally crossing the Pacific Ocean and back to your personal computer.

Now, it is time to actually send a picture of yourself around the world.

Navigate to the directory of the picture you want to send, and execute the following command:

$ scp -P 4455 <picture> root@<San Fransisco IP Address>:/

This command is normally used to copy the file <picture> from your local machine to the / directory of the San Fransisco server. However, since we are specifying -P 4455, which the San Fransisco server is forwarding to New York, the copy file command gets forwarded to the New York server instead. And since the New York server is forwarding port 4455, the chain goes on all the way up to Singapore which is configured to forward its port 4455 to your local personal computer’s port 22.

Your beautiful picture travelled from your computer to San Fransisco, New York, London, Amsterdam, Frankfurt, Singapore, and back to your computer.

Congratulations! You have officially gone around the world!

A quick time profile gives us the following results:

root@Razvans-MacBook-Pro Desktop $ time scp -P 4455 selfie.JPG [email protected]:/
selfie.JPG                       100%   19KB  19.4KB/s   00:00    

real	0m12.866s
user	0m0.019s
sys	0m0.009s

Alas in just 12.866 seconds you have travelled over 20,000 miles. You deserve a break.

]]> https://blog.razb.me/around-the-world-in-15-seconds/feed/ 1 48 https://blog.razb.me/networking-minecraft-on-an-inaccessible-server/ https://blog.razb.me/networking-minecraft-on-an-inaccessible-server/#respond Wed, 16 Mar 2016 06:02:28 +0000 http://razb.me/?p=28 Several months ago I was given access to a free, moderately powerful virtual private server (VPS) with the catch that the server sat behind a firewall which denied all incoming connections. The company hosting the VPS provided a web interface for accessing the server via a VNC connection which they routed internally. This was the only way to interact with the server.

Upon getting access to the VPS I immediately tried the first thing anyone would try, and that of course was running a Minecraft server on it. I was very excited about the possibility of hosting a completely free Minecraft server for my friends and I to enjoy, until I realized that it was impossible to connect to the server from the Internet.

So, I gave up and ran my Minecraft server on AWS instead. Thanks for reading.

Just kidding. I came across SSH reverse tunnels instead.

SSH is an incredibly powerful tool which is capable of doing a lot more than just being a secure shell to a remote machine. One of the additional features it has is reverse tunnelling.

Imagine you have two machines: MachineA, and MachineB.
MachineA can create outgoing connections, but can not accept incoming connections.
MachineB accepts both incoming and outgoing connections.

MachineA establishes an SSH connection to MachineB, and configures MachineB to forward all requests to port 8080 over to MachineA’s port 2020 (using the same SSH connection).

It looks something like this:

MachineB <—SSH—- MachineA // Establish the connection MachineB:8080 —SSH—> MachineA:2020 // Port forwarding over the established connection

This works fine because MachineA is not relying on any incoming connections made directly to it, but rather is able to accept incoming connections which arrive at MachineB. Now all I needed was another server which is not behind a firewall to act as MachineB.

Fortunately I have several small, inexpensive servers which were perfect for the job.

Using this functionality of SSH, I was able to piggy back one of my weaker servers to act as a proxy for the stronger server which was running Minecraft. My friends would connect to my weak server port 8080, and get forwarded over the SSH tunnel to the Minecraft server sitting behind a firewall on port 2020.

Alas, we were finally able to play Minecraft.

@implementation MyClass

+ (void)load {
    [[self sharedInstance] run];
}

+ (instancetype)sharedInstance {
    static MyClass *sharedClass;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        sharedClass = [self new];
    });
    return sharedClass;
}

- (void)run {
    // Do stuff
}

@end

Upon MyClass getting loaded into memory, +load is called by the runtime, which allocates, instantiates, and retains an instance of MyClass and then calls -run to do some arbitrary work. MyClass is not #imported from anywhere else, and it is also not retained by anyone else either. It’s there but you can’t see it.

In practice this is probably a bad idea because of the sneaky nature of things. So have a good reason before deciding to use something like this.