Bad experience with RedHat hands-on labs & exam.

Note: This is a translation of an article about the experience of a French guy following RedHat hands-on labs and taking the RHCE exam (source in French here).
This document was published on October the 12th 2016 and is under CC-BY-SA license.

I came across a LOT of technical and logistical/administrative problems when following the online RHCE training (RH299) and when taking the RHCE Exam (EX300F) in “individual exam” mode (in fact, I did not even take it).

Bugs in the RH299 training labs

(https://www.redhat.com/fr/services/training/rh299-rhce-certification-lab)

I chose to take this course “online”: I had access to the training materials for 3 months and to dedicated virtual machines for hands-on labs (with automated verification). The concept was really good and gave a lot of organizational flexibility. However, it was almost the same price as a “traditional” training in a classroom with an instructor.

The content of the training material was of good quality: clear explanations, appropriate practical work (apart from the videos that are not translated into French).
However, I came across a lot of bugs in the training labs.

I contacted the RedHat support about these problems (mid-March 2016). Someone quite competent replied point by point. I below summarizes these issues (the most important ones are at the beginning):

• Unable to use the AltGr key to enter special characters which are essential to hands-on labs: ‘{‘, ‘@’, ‘#’, etc. This problem was fortunately fixed in late March or early April.
• Error checking: the script that controls the outcome of the exercise is mistaken with respect to statement (for example: it searches for a connection named “review” when the statement says to create one named “exam”). So with a correct result in line with the statement, the tool still answers that this is wrong. I had the case on at least 3 assignments. RedHat confirms that it is definitely due to some translation “bugs”.
• Bad user interface: for example, if you quickly click twice on the button to stop a VM, it will unfortunately reset it without asking for any confirmation. You will lose all the work in progress.
• A lot of time wasted due to the use of VMs:
  For most hands-on labs, they ask to reset the VM before anything else. Except that it takes a few minutes to reset, and then 4 to 5 minutes to start, and then we must also configure the keyboard in French (it takes a long time, especially on first start). These are points that RedHat knows and attributes to their hosting infrastructure. They consider the possibility of reviewing their hands-on labs to avoid asking the reset.
  Each VM automatically stops after a timeout (the user can adjust it, but with a maximum). Except that the user is not informed when this timeout will be achieved. And when it is reached, the machine is abruptly stopped. RedHat told me that they are considering a pop-up to warn the user.
  And other technical problems were experienced such as the loss of network connectivity between the two VMs that serve the hands-on labs: they blamed the hosting infrastructure.
• Unable to copy/paste text into or from a VM. Again, they know it, but explain that the software they use does not allow this feature.
• After 4 hours of connection to the RedHat training site, the authentication cookie expires. Instead of asking the user to authenticate again, no feedback is given and all actions on the VMs are without effect anymore (and without error).
• Crashes in the nmtui tool (for configuring network connections), which prevent some hands-on labs to be carry out with this tool (it is still possible to do them with nmcli, but it is much less intuitive). This is probably a bug in nmtui, but I had never came across them outside of these VMs.

Suffice to say that this training was not for me as effective as it should have been.
A student shouldn’t have to debug the platform on which he is supposed to learn.

Many of these problems should have already been detected upstream by RedHat: it should have been enough to do some tests on the hands-on labs in a French configuration (keyboard, language, etc), before putting it online. They would have necessarily found that they could not type the necessary special characters and there were various bugs.

Note that the RedHat website at the end of May 2016 indicated that an update of this platform was underway: it is possible that some of these problems are now resolved (but I have no way to check it myself).

Finally, after ranting about RedHat about all these problems, I got they let me take the exam for free, as a gesture of goodwill (they also extended my access to the online training).

The EX300F exam

(https://www.redhat.com/en/services/training/ex300-red-hat-certified-engineer-rhce-exam)

Expected skills “Slightly out of date”

I noted several inconsistencies between the list of skills tested in the examination and the content of the RH299 training. In particular, the following topics were not covered in the training, and were yet displayed as required for the examination:

• Use /proc/sys and sysctl to modify and set kernel runtime parameters
• Produce and deliver reports on system utilization (processor, memory, disk, and network)
• (Network Services) Configure host-based and user-based security for the Service
• (HTTP) Configure private directories
• (HTTP) Configure group-managed pleased

After asking several people and called RedHat again on this subject, one of them finally told me specifically (mid-May 2016):

It appears that the information on the web page is slightly out of date.

The web page in question being the link above the “Objectives” tab.

In more detail, the RedHat‘s response was:

Unfortunately it appears that the information on the web page is slightly out of date. I will bring this to the attention of our web team. As a general answer to your question there are many factors that influence how well a given candidate will perform on the exam. Training alone is not usually sufficient to guarantee success but if you have relevant experience then training can make a difference. The other thing to consider is that RH299 is a rapid track course that is intended for experienced candidates.
The course covers material that we also cover in longer format in three separate four day courses. Going from twelve days of content to four days of content means that there are some topics that may not be covered or may not be covered in great depth.

Please see in-line for answers to some of the specific points you mention.
– Use /proc/sys and sysctl to modify and set kernel  runtime parameters
This is not an exam competency.

– Produce and deliver reports on system utilization (processor,memory, disk, and network)

This is not an exam competency per-se but this is the sort of information you might require in the course of performing other exam items and we expect candidates to be self-sufficient in this regard.

– (Network services) Configure host-based and user-based security for the service
This is a general competency. Some network services may support host-base or user-based security and we may ask you to implement access using these mechanisms.

– (HTTP) Configure private directories
– (HTTP) Configure group-managed content
Private directories are not an exam competency. We may request that you create directories that only allow certain users or user groups to manage content. I do not know if this is covered in RH299 but it is covered in RH254 (System Administration III).

Apparently the web page in question was amended in early June: no more objective on the sysctl topic and “Configure private directories” was renamed to “Configure access restrictions on directories” but the other objectives did not change.

Problems due to the “individual exam” mode

Small registration problems

I chose the “individual exam” mode (ex-kiosk) as exam dates in Lyon were all canceled (too few registrations). This method is much more flexible since you can select the date and time freely, which is very convenient. It is remotely monitored by a webcam.

RedHat goes through the same company as the Linux Foundation: ExamsLocal (Innovative Exams). However, unlike the Linux Foundation that allows to take the exam from home, it must be conducted in one of the RedHat examination centers (only Paris in France). I guess they consider it more secure, but it’s much less convenient (and more expensive) for those who don’t live in the Paris area.

During registration, available dates were changing all the time: at the beginning of my browsing on the registration site (managed by ExamsLocal), the date that interested me was available and then after a few minutes all schedules with that date became unavailable, then available again after a few minutes. Bug? I don’t know: I booked my date as soon as possible, as a precaution.

I received an immediate confirmation by email, but it gave the time in another timezone (UTC-1): not very convenient. Fortunately, the following emails were well in the French timezone.

Keyboard problem

On the day, after blocking 3 days of study, I went to Paris (no chance, railway strike, but ultimately not much trouble).

The exam went on a dedicated PC, where everything is locked so you can’t cheat. You have to log on to ExamsLocal (as expected) to take the exam.

Problem: we were faced with a AZERTY keyboard, but the OS was configured for QWERTY and no access to system settings to change it. In short, you had to type your password, translating positions from AZERTY to QWERTY. As a security aware sysadmin, I had special characters in my password, I had a hard time and typed my password in the login field before copying/pasting in the password field (I couldn’t find a better way).

Note: RedHat assured me later that this problem has now been fixed, which seems confirmed by the “Azerty keyboard” mention when seeking a session of this type.

SSO login problem that prevents the exam from being taken

And here it is: unable to log if you have a login from the ExamsLocal Linux Foundation.

Indeed, during the RHCE registration, I mentioned the ExamsLocal account, previously used when passing the Linux Foundation exams: as I was already identified with it, I didn’t need to create a new account. There were no warnings on this in RedHat or ExamsLocal websites. The ExamsLocal website correctly added the exam to my account and sent me reminders by email before the exam, etc: everything seemed in order.

Except that the PC used to take the exam in “individual exam” mode was locked with forbidden access to the Internet. ExamsLocal website uses a Single-Sign-On procedure provided by the Linux Foundation website: in this case, connection was redirected to a page on the Linux Foundation website… that was blocked by the system.

At StartX (the company that manages the RedHat exams in Paris), they told me that it had happened once, and they had reported it to RedHat. But RedHat did not correct the problem they already knew neither changed its website (to inform people about this problem), modified the filter rules in their PC or warned people about this issue (in my case, nobody let me know).

StartX team told me that I had to create another account on the ExamsLocal website (without SSO) and my exam would be transferred. Yes, except that they can’t do it themselves: this must go through ExamsLocal, which are in the United States, and there should be a period of “a few days” (wait later you will laugh). Joy, happiness. I could not take the exam because of this and went home very angry.

Obviously, I could log on the examslocal website with my personal PC (I carried it with me at StartX), and I could have taken the exam with it. But no, RedHat’s refusal: only the RedHat dedicated PC is allowed for the exam. On the other hand, StartX people have no access to the PC used for the examination.

Icing on the cake that evening, I received an email from RedHat who told me I failed my exam, because I only had 46 points, while it was 210 in order to pass it: how could I have 46 points without taking the exam? I asked several people from RedHat, but none replied.

Over 4 months to take into account some of my complaints

I immediately demanded compensation and the opportunity to retake the exam. Through the only means of communication that I had: a claim form from the RedHat website. It was the beginning of an obstacle race.

No news after 2 weeks (except for a “Please allow me to Investigate the box and will get back to you soon”), I asked again. Still no response after 3 weeks, so I sent a letter with recorded delivery. Then I got a call from a person from RedHat London who apologized for all these failures, assured me that the issue was taken very seriously by the “top management” and I would be quickly informed. He gave me his direct phone number. I was somewhat reassured (I was wrong).

A few days later, I received an email from another person from RedHat, who apologized for forgetting to reply to my request (!). This email said that, as compensation, RedHat would offer me a free retake of the EX300 exam and another of my choice. 2 exams to be taken before the beginning of December, 2016 (we were early July) …

After discussions with my interlocutor from London and another one (I’ll spare you the wasted time and ping pong games between these interlocutors), the final position of RedHat was:

• Reimbursement of transportation related to the exam that I could not take (only on proof)
• Ability to retake the exam for free in the next 12 months
• And, as compensation, the possibility of taking another exam of my choice in the next 12 months

RedHat seems to think that a free exam is a nice gift as it is worth several hundred euros. This may be true for a person who needs additional certifications. In my case, I would probably do nothing of this. Especially since, after all these problems, it is not the first thing I want … If I do not actually make use of it, I will try to offer it to someone else (but I have nobody around me currently interested in).

I had rather sought financial compensation for days spent preparing for the exam (I’ll have to prepare again, given the time), and to take it. The answer was clearly no.

A month later, I still had nothing. So I sent a 2nd letter with recorded delivery, call my interlocutor from London and got an email confirming that I will receive very soon what had been promised.

It took a total of over 3 months, 2 letters with recorded delivery, much time and energy lost, to get the codes to take the exam again.

To date, more than 4 months after the initial exam, I still have not been reimbursed for transportation costs (I’ll spare you the many reminders and “internal procedures” to be followed for a simple transfer) .

Initially, I wanted to wait to be reimbursed before publishing this article (not to influence the procedure in progress, whether in one direction or another). But never mind, it seems to me that the exam candidates need to be aware of the SSO login problem, not to suffer the same setbacks as me.

In the end, I intend to take the RHCE exam, I will not give up. But, given the time elapsed, I must restart my study from scratch. And I think I need a break after all that: it will be for later 😉

RedHat portal and its invisible conditions

(https://www.redhat.com/wapps/ugc/protected/account.html)

Besides the above problems, this one is much less serious, but equally symptomatic.

From this portal, once authenticated, you can access the list of the various certifications.

Problem: when you click on “Certifications”, RedHat asks you to read and validate “Logos and Guidelines – Terms and Conditions” with a link to read these conditions … this doesn’t redirect to the right place. In short, RedHat asks you to validate conditions that you can’t read.

I first reported this problem in mid-December 2015. At the time, the link showed a downright “Proxy Error”.

The “Proxy Error” has been fixed mid-January 2016, but the link now returns to https://www.redhat.com/rhtapps/certification/… I still don’t see the “terms and conditions” to read.

Late February 2016, RedHat sent me an email with a draft of these conditions:

We’re in the process of rewriting these, but here is what we currently have. Please see attachment.

To date (mid-October 2016), despite my numerous reminders, the bug is still present, and I don’t know how to read the final conditions RedHat asks me to confirm.

What to think about this?

I haven’t any doubt about the RedHat employees’ skills nor the architectural complexity of RedHat websites, or the difficulty of implementing secure online exams, but it’s quite ironic that all these technical problems occur with an editor certifying system administrators …

I mainly see the result of administrative burdens, lack of communication between services, complexities of outsourcing and a dilution of responsibility. This kind of symptom unfortunately occurs in many large companies. But, at such a level, I admit that I would not have imagined that.

I also think that the methods that I have chosen (online training and individual examination) are not frequently used and may not have received the same attention as the more “classical” modes. That doesn’t excuse anything, of course …

In the end, as you could imagine, I’m not fully pleased with this experience …